{ "version": "1.0.0", "exported_at": "2026-06-01T05:10:00.000Z", "project": { "name": "Gumtree Scraper Real Estate Listing", "description": "Octoparse-equivalent Gumtree Australia real estate listing scraper. Extracts Gumtree property/ad detail page URLs and Page_Number from https://www.gumtree.com.au/s-flatshare-houseshare/c18294. Pagination is implemented with a next-page condition loop and CSV append mode so all reachable listing pages are accumulated. Because attached analysis showed Gumtree Access Denied / HTTP 403 on sample detail pages, this best-effort template uses robust JavaScript extraction from anchors and embedded HTML; it may still require a trusted browser profile, clean session, or residential IP if Gumtree blocks access.", "color": "bg-[#4589ff]", "template_id": "ai-generated" }, "blocks": [ { "block_id": "navigate-1", "block_type": "process", "title": "Navigate", "description": "Go to a URL", "position_x": 120, "position_y": 260, "config": { "url": "https://www.gumtree.com.au/s-flatshare-houseshare/c18294", "color": "bg-[#4589ff]" } }, { "block_id": "wait-for-page-load-1", "block_type": "process", "title": "Wait for Page Load", "description": "Wait for page to finish loading", "position_x": 480, "position_y": 260, "config": { "timeout": 30 } }, { "block_id": "wait-for-element-1", "block_type": "process", "title": "Wait for Element", "description": "Wait until element appears", "position_x": 840, "position_y": 260, "config": { "selector": "body", "timeout": 30, "visible": true } }, { "block_id": "text-contains-1", "block_type": "process", "title": "Text Contains", "description": "Check if page contains text", "position_x": 1200, "position_y": 260, "config": { "text": "Access Denied", "selector": "body", "caseSensitive": false } }, { "block_id": "end-1", "block_type": "output", "title": "End", "description": "Terminate execution flow", "position_x": 1560, "position_y": 260, "config": {} }, { "block_id": "inject-javascript-1", "block_type": "process", "title": "Inject JavaScript", "description": "Execute custom JavaScript", "position_x": 1200, "position_y": 580, "config": { "jsCode": "(() => {\n const pageNumber = (() => {\n const href = location.href;\n const path = location.pathname;\n const m = path.match(/\\/page-(\\d+)(?:\\/|$)/i) || href.match(/[?&](?:page|p)=(\\d+)/i);\n return m ? m[1] : '1';\n })();\n\n document.querySelectorAll('.uscraper-gumtree-row').forEach(el => el.remove());\n document.querySelectorAll('[data-uscraper-next=\"true\"]').forEach(el => el.removeAttribute('data-uscraper-next'));\n\n const urls = new Map();\n const addUrl = (raw) => {\n if (!raw) return;\n let value = String(raw)\n .replace(/\\\\u002[Ff]/g, '/')\n .replace(/\\\\\\//g, '/')\n .replace(/&/g, '&')\n .trim();\n if (!value || !value.includes('/s-ad/')) return;\n try {\n const u = new URL(value, location.origin);\n if (!/gumtree\\.com\\.au$/i.test(u.hostname.replace(/^www\\./i, ''))) return;\n if (!/\\/s-ad\\//i.test(u.pathname)) return;\n const id = (u.pathname.match(/\\/(\\d+)\\/?$/) || [null, u.origin + u.pathname])[1];\n u.hash = '';\n urls.set(id, u.href);\n } catch (e) {}\n };\n\n document.querySelectorAll('a[href]').forEach(a => addUrl(a.getAttribute('href')));\n\n const html = document.documentElement.innerHTML\n .replace(/\\\\u002[Ff]/g, '/')\n .replace(/\\\\\\//g, '/')\n .replace(/&/g, '&');\n\n const absoluteRe = /https?:\\/\\/(?:www\\.)?gumtree\\.com\\.au\\/s-ad\\/[^\"'<>\\s]+?\\/\\d+/gi;\n const relativeRe = /(?:^|[\"'\\s(])((?:\\/s-ad\\/)[^\"'<>\\s]+?\\/\\d+)/gi;\n\n let match;\n while ((match = absoluteRe.exec(html)) !== null) addUrl(match[0]);\n while ((match = relativeRe.exec(html)) !== null) addUrl(match[1]);\n\n const container = document.createElement('div');\n container.id = 'uscraper-gumtree-results';\n container.style.display = 'block';\n container.style.position = 'relative';\n container.style.zIndex = '-1';\n container.style.height = '1px';\n container.style.overflow = 'hidden';\n\n for (const url of urls.values()) {\n const row = document.createElement('div');\n row.className = 'uscraper-gumtree-row';\n row.setAttribute('data-url', url);\n row.setAttribute('data-page-number', pageNumber);\n row.textContent = url;\n container.appendChild(row);\n }\n document.body.appendChild(container);\n\n const currentPage = parseInt(pageNumber, 10) || 1;\n const isDisabled = (el) => {\n const cls = (el.className || '').toString().toLowerCase();\n return el.disabled || el.getAttribute('aria-disabled') === 'true' || cls.includes('disabled') || cls.includes('inactive');\n };\n const isVisible = (el) => {\n const style = window.getComputedStyle(el);\n const rect = el.getBoundingClientRect();\n return style.display !== 'none' && style.visibility !== 'hidden' && rect.width > 0 && rect.height > 0;\n };\n const nextCandidates = Array.from(document.querySelectorAll('a[href], button')).filter(el => {\n if (isDisabled(el)) return false;\n const text = (el.textContent || '').trim().toLowerCase();\n const aria = (el.getAttribute('aria-label') || '').toLowerCase();\n const title = (el.getAttribute('title') || '').toLowerCase();\n const rel = (el.getAttribute('rel') || '').toLowerCase();\n const testid = (el.getAttribute('data-testid') || '').toLowerCase();\n const href = el.getAttribute('href') || '';\n let hrefPage = null;\n const hrefMatch = href.match(/\\/page-(\\d+)(?:\\/|$)/i) || href.match(/[?&](?:page|p)=(\\d+)/i);\n if (hrefMatch) hrefPage = parseInt(hrefMatch[1], 10);\n return rel === 'next' || aria.includes('next') || title.includes('next') || testid.includes('next') || text === 'next' || text === '›' || text === '>' || text.includes('next') || hrefPage === currentPage + 1;\n });\n\n const next = nextCandidates.find(isVisible) || nextCandidates[0];\n if (next) next.setAttribute('data-uscraper-next', 'true');\n\n return { rows: urls.size, pageNumber, hasNext: Boolean(next) };\n})();", "waitForCompletion": true, "timeout": 10 } }, { "block_id": "element-exists-2", "block_type": "process", "title": "Element Exists", "description": "Check if element exists", "position_x": 1560, "position_y": 580, "config": { "selector": ".uscraper-gumtree-row" } }, { "block_id": "structured-export-1", "block_type": "process", "title": "Structured Export", "description": "Export data with custom columns", "position_x": 1920, "position_y": 580, "config": { "rowSelector": ".uscraper-gumtree-row", "fileName": "gumtree-scraper-real-estate-listing.csv", "saveLocation": "C:\\Users\\theskd\\Documents\\UScraper\\templates", "includeHeaders": true, "fileMode": "append", "columns": [ { "name": "URL", "selector": "", "attribute": "data-url" }, { "name": "Page_Number", "selector": "", "attribute": "data-page-number" } ] } }, { "block_id": "end-2", "block_type": "output", "title": "End", "description": "Terminate execution flow", "position_x": 1560, "position_y": 900, "config": {} }, { "block_id": "element-exists-1", "block_type": "process", "title": "Element Exists", "description": "Check if element exists", "position_x": 2280, "position_y": 580, "config": { "selector": "[data-uscraper-next=\"true\"]" } }, { "block_id": "end-3", "block_type": "output", "title": "End", "description": "Terminate execution flow", "position_x": 2280, "position_y": 900, "config": {} }, { "block_id": "inject-javascript-2", "block_type": "process", "title": "Inject JavaScript", "description": "Execute custom JavaScript", "position_x": 2640, "position_y": 900, "config": { "jsCode": "(() => {\n const next = document.querySelector('[data-uscraper-next=\"true\"]');\n if (!next) return false;\n if (next.tagName && next.tagName.toLowerCase() === 'a' && next.href) {\n location.href = next.href;\n } else {\n next.click();\n }\n return true;\n})();", "waitForCompletion": true, "timeout": 10 } }, { "block_id": "wait-for-page-load-2", "block_type": "process", "title": "Wait for Page Load", "description": "Wait for page to finish loading", "position_x": 3000, "position_y": 900, "config": { "timeout": 30 } }, { "block_id": "sleep-1", "block_type": "process", "title": "Sleep", "description": "Wait for specified time", "position_x": 3360, "position_y": 900, "config": { "duration": 2 } } ], "connections": [ { "from_block_id": "navigate-1", "from_connector_id": "right", "to_block_id": "wait-for-page-load-1", "to_connector_id": "left" }, { "from_block_id": "wait-for-page-load-1", "from_connector_id": "right", "to_block_id": "wait-for-element-1", "to_connector_id": "left" }, { "from_block_id": "wait-for-element-1", "from_connector_id": "right", "to_block_id": "text-contains-1", "to_connector_id": "left" }, { "from_block_id": "text-contains-1", "from_connector_id": "true", "to_block_id": "end-1", "to_connector_id": "left" }, { "from_block_id": "text-contains-1", "from_connector_id": "false", "to_block_id": "inject-javascript-1", "to_connector_id": "left" }, { "from_block_id": "inject-javascript-1", "from_connector_id": "right", "to_block_id": "element-exists-2", "to_connector_id": "left" }, { "from_block_id": "element-exists-2", "from_connector_id": "true", "to_block_id": "structured-export-1", "to_connector_id": "left" }, { "from_block_id": "element-exists-2", "from_connector_id": "false", "to_block_id": "end-2", "to_connector_id": "left" }, { "from_block_id": "structured-export-1", "from_connector_id": "right", "to_block_id": "element-exists-1", "to_connector_id": "left" }, { "from_block_id": "element-exists-1", "from_connector_id": "true", "to_block_id": "inject-javascript-2", "to_connector_id": "left" }, { "from_block_id": "element-exists-1", "from_connector_id": "false", "to_block_id": "end-3", "to_connector_id": "left" }, { "from_block_id": "inject-javascript-2", "from_connector_id": "right", "to_block_id": "wait-for-page-load-2", "to_connector_id": "left" }, { "from_block_id": "wait-for-page-load-2", "from_connector_id": "right", "to_block_id": "sleep-1", "to_connector_id": "left" }, { "from_block_id": "sleep-1", "from_connector_id": "right", "to_block_id": "wait-for-element-1", "to_connector_id": "left" } ], "canvas_elements": [ { "id": "group-load", "element_type": "group", "title": "Page Load", "color": "#08bdba", "position_x": 48, "position_y": 156, "width": 3560, "height": 936, "z_index": 20, "data": { "memberBlockIds": [ "navigate-1", "wait-for-page-load-1", "wait-for-element-1", "wait-for-page-load-2", "sleep-1" ] } }, { "id": "group-pagination", "element_type": "group", "title": "Pagination Loop", "color": "#ff832b", "position_x": 1128, "position_y": 156, "width": 1400, "height": 616, "z_index": 20, "data": { "memberBlockIds": [ "text-contains-1", "element-exists-2", "element-exists-1" ] } }, { "id": "group-control", "element_type": "group", "title": "Control Flow", "color": "#8d8d8d", "position_x": 1488, "position_y": 156, "width": 1040, "height": 936, "z_index": 20, "data": { "memberBlockIds": [ "end-1", "end-2", "end-3" ] } }, { "id": "group-interaction", "element_type": "group", "title": "Interaction", "color": "#a56eff", "position_x": 1128, "position_y": 476, "width": 1760, "height": 616, "z_index": 20, "data": { "memberBlockIds": [ "inject-javascript-1", "inject-javascript-2" ] } }, { "id": "group-extract", "element_type": "group", "title": "Data Extraction", "color": "#42be65", "position_x": 1848, "position_y": 476, "width": 380, "height": 296, "z_index": 20, "data": { "memberBlockIds": [ "structured-export-1" ] } }, { "id": "note-overview", "element_type": "note", "title": "Overview", "content": "Octoparse-equivalent Gumtree Australia real estate listing scraper. Extracts Gumtree property/ad detail page URLs and Page_Number from https://www.gumtree.com.au/s-flatshare-houseshare/c18294. Pagination is implemented with a next-page condition loop and CSV append mode so all reachable listing pages are accumulated. Because attached analysis showed Gumtree Access Denied / HTTP 403 on sample detail pages, this best-effort template uses robust JavaScript extraction from anchors and embedded HTML; it may still require a trusted browser profile, clean session, or residential IP if Gumtree blocks access.", "color": "#f1c21b", "position_x": 80, "position_y": 20, "width": 480, "height": 160, "z_index": 22, "data": {} }, { "id": "note-block-text-contains-1", "element_type": "note", "title": "Note: Text Contains", "content": "Condition block: checks `body`. True / False branches control which path runs next. Keep enough space between branches so both connector lines are visible.", "color": "#ee5396", "position_x": 1400, "position_y": 240, "width": 340, "height": 131, "z_index": 22, "data": { "block_id": "text-contains-1" } }, { "id": "note-block-inject-javascript-1", "element_type": "note", "title": "Note: Inject JavaScript", "content": "Runs custom JavaScript in the page: `(() => {\n const pageNumber = (() => {\n const href = location.href;\n const path = location.pat...` Verify in browser if results are empty.", "color": "#ee5396", "position_x": 1400, "position_y": 560, "width": 340, "height": 140, "z_index": 22, "data": { "block_id": "inject-javascript-1" } }, { "id": "note-block-element-exists-2", "element_type": "note", "title": "Note: Element Exists", "content": "Condition block: checks `.uscraper-gumtree-row`. True / False branches control which path runs next. Keep enough space between branches so both connector lines are visible.", "color": "#ee5396", "position_x": 1760, "position_y": 560, "width": 340, "height": 137, "z_index": 22, "data": { "block_id": "element-exists-2" } }, { "id": "note-block-structured-export-1", "element_type": "note", "title": "Note: Structured Export", "content": "Extracts rows matching `.uscraper-gumtree-row`. Confirm row count > 0 before running at scale.", "color": "#ee5396", "position_x": 2120, "position_y": 560, "width": 340, "height": 111, "z_index": 22, "data": { "block_id": "structured-export-1" } }, { "id": "note-block-element-exists-1", "element_type": "note", "title": "Note: Element Exists", "content": "Condition block: checks `[data-uscraper-next=\"true\"]`. True / False branches control which path runs next. Keep enough space between branches so both connector lines are visible.", "color": "#ee5396", "position_x": 2480, "position_y": 560, "width": 340, "height": 139, "z_index": 22, "data": { "block_id": "element-exists-1" } }, { "id": "note-block-inject-javascript-2", "element_type": "note", "title": "Note: Inject JavaScript", "content": "Runs custom JavaScript in the page: `(() => {\n const next = document.querySelector('[data-uscraper-next=\"true\"]');\n if (!next) return f...` Verify in browser if results are empty.", "color": "#ee5396", "position_x": 2840, "position_y": 880, "width": 340, "height": 140, "z_index": 22, "data": { "block_id": "inject-javascript-2" } } ] }