{ "version": "1.0.0", "exported_at": "2026-05-31T00:00:00.000Z", "project": { "name": "TikTok Scraper", "description": "Best-effort TikTok public profile/video scraper. Default URL targets https://www.tiktok.com/@zachking because TikTok's official @tiktok profile returned profile stats but zero accessible post rows in the automated test session. Edit the Navigate URL for another public TikTok profile. The workflow dismisses popups, waits for the page, scrolls for dynamic/infinite-scroll content, scans rendered DOM and embedded JSON, then uses TikTok same-origin web API fallback endpoints to collect paginated post/video metadata. Exports video URL, creator, caption/title, hashtags, thumbnail, views, likes, comments, shares, bookmarks, post date, duration, and profile-level following/followers/likes. TikTok may still block automation with login/CAPTCHA/region/anti-bot checks; if no video data is accessible, a diagnostic row is exported.", "color": "bg-[#4589ff]", "template_id": "ai-generated" }, "blocks": [ { "block_id": "navigate-1", "block_type": "process", "title": "Navigate", "description": "Go to a URL", "position_x": 120, "position_y": 260, "config": { "url": "https://www.tiktok.com/@zachking", "color": "bg-[#4589ff]" } }, { "block_id": "wait-for-page-load-1", "block_type": "process", "title": "Wait for Page Load", "description": "Wait for page to finish loading", "position_x": 480, "position_y": 260, "config": { "timeout": 45 } }, { "block_id": "inject-javascript-1", "block_type": "process", "title": "Inject JavaScript", "description": "Execute custom JavaScript", "position_x": 840, "position_y": 260, "config": { "jsCode": "Array.from(document.querySelectorAll('button, [role=\"button\"]')).find(el => /^(accept|accept all|allow all|agree|got it|ok|continue|not now)$/i.test((el.innerText || el.textContent || '').trim()))?.click(); Array.from(document.querySelectorAll('button[aria-label*=\"Close\" i], [role=\"button\"][aria-label*=\"Close\" i]')).forEach(el => el.click());", "waitForCompletion": true, "timeout": 10 } }, { "block_id": "wait-for-element-1", "block_type": "process", "title": "Wait for Element", "description": "Wait until element appears", "position_x": 1200, "position_y": 260, "config": { "selector": "body", "timeout": 30, "visible": true } }, { "block_id": "inject-javascript-2", "block_type": "process", "title": "Inject JavaScript", "description": "Execute custom JavaScript", "position_x": 1560, "position_y": 260, "config": { "jsCode": "return new Promise(resolve => {\n const sleep = ms => new Promise(r => setTimeout(r, ms));\n const txt = el => (el?.innerText || el?.textContent || '').trim();\n const clean = v => (v == null ? '' : String(v));\n const abs = u => { try { return new URL(u, location.href).href; } catch { return u || ''; } };\n const uniqueId = decodeURIComponent((location.pathname.match(/@([^/?#]+)/) || [])[1] || '');\n const items = new Map();\n const diagnostics = [];\n\n function dismiss() {\n Array.from(document.querySelectorAll('button, [role=\"button\"]')).find(el => /^(accept|accept all|allow all|agree|got it|ok|continue|not now)$/i.test(txt(el)))?.click();\n Array.from(document.querySelectorAll('button[aria-label*=\"Close\" i], [role=\"button\"][aria-label*=\"Close\" i], svg[aria-label*=\"Close\" i]')).forEach(el => { try { el.click(); } catch {} });\n }\n\n function addItem(raw) {\n if (!raw) return;\n const stats = raw.stats || raw.statistics || raw.statsV2 || raw.statsV2 || {};\n const author = raw.author || raw.authorInfo || raw.authorStats || {};\n const video = raw.video || raw.videoInfo || {};\n const id = clean(raw.id || raw.itemId || raw.aweme_id || raw.videoId || raw.video_id || raw.videoIdStr);\n let creator = clean(raw.creator || raw.authorId || raw.authorUniqueId || raw.uniqueId || author.uniqueId || author.unique_id || author.id || uniqueId);\n creator = decodeURIComponent(creator.replace(/^@/, ''));\n let url = clean(raw.video_url || raw.url || raw.href || raw.shareUrl || raw.share_url);\n if (!url && creator && id) url = location.origin + '/@' + creator + '/video/' + id;\n if (url && !/^https?:/i.test(url)) url = abs(url);\n const caption = clean(raw.title || raw.caption || raw.desc || raw.description || raw.text);\n const tagSource = [caption, clean(raw.hashtags)].join(' ');\n const hashtags = clean(raw.hashtags) || Array.from(new Set(tagSource.match(/#[\\p{L}\\p{N}_]+/gu) || [])).join(' ');\n const cover = clean(raw.thumbnail_url || raw.cover || raw.image || video.cover || video.dynamicCover || video.originCover || (Array.isArray(video.coverUrlList) ? video.coverUrlList[0] : '') || (Array.isArray(video.dynamicCoverUrlList) ? video.dynamicCoverUrlList[0] : ''));\n const key = url || id || caption;\n if (!key) return;\n const existing = items.get(key) || {};\n items.set(key, {\n row_type: 'video',\n video_url: existing.video_url || url,\n video_id: existing.video_id || id,\n creator: existing.creator || creator,\n profile_url: existing.profile_url || (creator ? location.origin + '/@' + creator : ''),\n title_or_caption: existing.title_or_caption || caption,\n hashtags: existing.hashtags || hashtags,\n thumbnail_url: existing.thumbnail_url || cover,\n views: existing.views || clean(raw.views || raw.playCount || raw.play_count || stats.playCount || stats.play_count),\n likes: existing.likes || clean(raw.likes || raw.diggCount || raw.like_count || stats.diggCount || stats.digg_count),\n comments: existing.comments || clean(raw.comments || raw.commentCount || raw.comment_count || stats.commentCount || stats.comment_count),\n shares: existing.shares || clean(raw.shares || raw.shareCount || raw.share_count || stats.shareCount || stats.share_count),\n bookmarks: existing.bookmarks || clean(raw.bookmarks || raw.collectCount || raw.collect_count || stats.collectCount || stats.collect_count),\n post_date: existing.post_date || clean(raw.post_date || raw.createTime || raw.create_time),\n video_duration: existing.video_duration || clean(raw.duration || raw.video_duration || video.duration)\n });\n }\n\n function addFromDom() {\n document.querySelectorAll('a[href*=\"/video/\"]').forEach(a => {\n const card = a.closest('div[data-e2e=\"user-post-item\"], div[data-e2e=\"search-card-item\"], article, li, div') || a;\n const img = card.querySelector('img') || a.querySelector('img');\n const viewEl = card.querySelector('[data-e2e=\"video-views\"], [data-e2e*=\"view\"], strong');\n addItem({\n href: a.href || a.getAttribute('href'),\n title: (img?.getAttribute('alt') || a.getAttribute('aria-label') || a.getAttribute('title') || txt(card)).replace(/^.*?on TikTok:?\\s*/i, '').trim(),\n thumbnail_url: img?.src || img?.getAttribute('src') || '',\n views: txt(viewEl)\n });\n });\n }\n\n function scanJson(obj, depth = 0) {\n if (!obj || depth > 10) return;\n if (Array.isArray(obj)) { obj.forEach(x => scanJson(x, depth + 1)); return; }\n if (typeof obj !== 'object') return;\n if (obj.itemStruct) addItem(obj.itemStruct);\n if (obj.itemList && Array.isArray(obj.itemList)) obj.itemList.forEach(addItem);\n if (obj.items && Array.isArray(obj.items)) obj.items.forEach(addItem);\n if (obj.ItemModule && typeof obj.ItemModule === 'object') Object.values(obj.ItemModule).forEach(addItem);\n const hasVideoShape = obj.id && (obj.desc || obj.description || obj.stats || obj.statistics || obj.video || obj.author);\n if (hasVideoShape) addItem(obj);\n Object.keys(obj).slice(0, 140).forEach(k => scanJson(obj[k], depth + 1));\n }\n\n function scanScriptsAndHtml() {\n let html = document.documentElement.innerHTML || '';\n html = html.replace(/\\\\u002F/g, '/').replace(/\\\\\\//g, '/').replace(/"/g, '\"').replace(/&/g, '&');\n const urls = new Set([...(html.match(/https?:\\/\\/(?:www\\.)?tiktok\\.com\\/@[^\\s\"'<>]+\\/video\\/\\d+/g) || []), ...(html.match(/\\/@[A-Za-z0-9._-]+\\/video\\/\\d+/g) || []).map(u => location.origin + u)]);\n urls.forEach(u => addItem({ href: u }));\n document.querySelectorAll('script').forEach(s => {\n const raw = s.textContent || '';\n if (!raw || raw.length < 20) return;\n if (raw.includes('itemStruct') || raw.includes('ItemModule') || raw.includes('itemList') || raw.includes('playCount') || raw.includes('/video/')) {\n try { scanJson(JSON.parse(raw)); } catch {}\n }\n });\n try { scanJson(window.SIGI_STATE); } catch {}\n try { scanJson(window.__UNIVERSAL_DATA_FOR_REHYDRATION__); } catch {}\n try { scanJson(window.__INIT_PROPS__); } catch {}\n }\n\n async function fetchJson(url) {\n const res = await fetch(url, { credentials: 'include', headers: { 'accept': 'application/json, text/plain, */*' } });\n const body = await res.text();\n try { return JSON.parse(body); } catch { return { __status: res.status, __text: body.slice(0, 500) }; }\n }\n\n function getCookie(name) {\n return (document.cookie.split('; ').find(x => x.startsWith(name + '=')) || '').split('=').slice(1).join('=');\n }\n\n function baseParams(extra) {\n const webId = localStorage.getItem('wid') || getCookie('ttwid') || String(Math.floor(Math.random() * 1e18));\n const p = new URLSearchParams({\n WebIdLastTime: String(Math.floor(Date.now() / 1000) - 100),\n aid: '1988',\n app_language: 'en',\n app_name: 'tiktok_web',\n browser_language: navigator.language || 'en-US',\n browser_name: 'Mozilla',\n browser_online: String(navigator.onLine),\n browser_platform: navigator.platform || 'Win32',\n browser_version: navigator.userAgent,\n channel: 'tiktok_web',\n cookie_enabled: String(navigator.cookieEnabled),\n device_id: webId.replace(/\\D/g, '').slice(0, 19) || String(Math.floor(Math.random() * 1e18)),\n device_platform: 'web_pc',\n focus_state: 'true',\n from_page: 'user',\n history_len: '2',\n is_fullscreen: 'false',\n is_page_visible: 'true',\n language: 'en',\n os: 'windows',\n priority_region: '',\n referer: location.href,\n region: 'US',\n screen_height: String(screen.height || 1080),\n screen_width: String(screen.width || 1920),\n tz_name: Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC',\n webcast_language: 'en'\n });\n const msToken = getCookie('msToken');\n if (msToken) p.set('msToken', msToken);\n Object.entries(extra || {}).forEach(([k, v]) => p.set(k, String(v)));\n return p.toString();\n }\n\n async function fetchApiFallback() {\n let secUid = '';\n let userId = '';\n try {\n const detail = await fetchJson(location.origin + '/api/user/detail/?' + baseParams({ uniqueId }));\n scanJson(detail);\n secUid = detail?.userInfo?.user?.secUid || detail?.userInfo?.user?.sec_uid || detail?.user?.secUid || '';\n userId = clean(detail?.userInfo?.user?.id || detail?.user?.id || '');\n if (!secUid) secUid = ((document.documentElement.innerHTML.match(/\"secUid\"\\s*:\\s*\"([^\"]+)\"/) || [])[1] || '').replace(/\\\\u002F/g, '/');\n diagnostics.push('user_detail_status=' + clean(detail?.statusCode || detail?.status_code || detail?.__status || 'ok') + '; secUid=' + (secUid ? 'found' : 'missing') + '; userId=' + (userId || 'missing'));\n } catch (e) {\n diagnostics.push('user_detail_error=' + e.message);\n }\n\n const endpointCandidates = [\n '/api/post/item_list/',\n '/api/user/posts/',\n '/api/user/item_list/'\n ];\n\n for (const endpoint of endpointCandidates) {\n let cursor = '0';\n for (let page = 0; page < 6; page++) {\n try {\n const extra = { count: '35', cursor };\n if (secUid) extra.secUid = secUid;\n if (userId) extra.userId = userId;\n if (uniqueId) extra.uniqueId = uniqueId;\n const data = await fetchJson(location.origin + endpoint + '?' + baseParams(extra));\n scanJson(data);\n const list = data?.itemList || data?.items || data?.data?.itemList || data?.data?.items || [];\n if (Array.isArray(list)) list.forEach(addItem);\n diagnostics.push(endpoint + '_page_' + page + '_items=' + (Array.isArray(list) ? list.length : 0) + '_status=' + clean(data?.statusCode || data?.status_code || data?.__status || 'ok'));\n const nextCursor = clean(data?.cursor || data?.nextCursor || data?.maxCursor || data?.data?.cursor || data?.data?.nextCursor || '');\n const hasMore = data?.hasMore === true || data?.hasMore === 1 || data?.has_more === true || data?.has_more === 1 || data?.data?.hasMore === true || data?.data?.hasMore === 1;\n if (!hasMore || !nextCursor || nextCursor === cursor) break;\n cursor = nextCursor;\n await sleep(700);\n } catch (e) {\n diagnostics.push(endpoint + '_error=' + e.message);\n break;\n }\n }\n if (items.size > 0) break;\n }\n }\n\n async function scrollPass() {\n await sleep(3500);\n let lastHeight = 0;\n let lastCount = 0;\n let stable = 0;\n for (let i = 0; i < 18; i++) {\n dismiss();\n addFromDom();\n scanScriptsAndHtml();\n const h = Math.max(document.body.scrollHeight || 0, document.documentElement.scrollHeight || 0);\n const c = items.size + document.querySelectorAll('a[href*=\"/video/\"]').length;\n if (h === lastHeight && c === lastCount) stable++; else stable = 0;\n lastHeight = h;\n lastCount = c;\n window.scrollTo(0, h);\n if (stable >= 5 && i >= 7) break;\n await sleep(1200);\n }\n }\n\n function buildRows() {\n const old = document.querySelector('#uscraper-tiktok-rows');\n if (old) old.remove();\n const wrap = document.createElement('div');\n wrap.id = 'uscraper-tiktok-rows';\n wrap.style.display = 'none';\n\n const profileFollowing = txt(document.querySelector('[data-e2e=\"following-count\"]'));\n const profileFollowers = txt(document.querySelector('[data-e2e=\"followers-count\"]'));\n const profileLikes = txt(document.querySelector('[data-e2e=\"likes-count\"]'));\n\n let arr = Array.from(items.values()).filter(x => x.video_url || x.video_id || x.title_or_caption);\n if (arr.length === 0) {\n arr = [{\n row_type: 'diagnostic_no_video_rows',\n video_url: '',\n video_id: '',\n creator: uniqueId,\n profile_url: location.origin + '/@' + uniqueId,\n title_or_caption: 'No TikTok video rows found. TikTok may have returned a login, CAPTCHA, regional block, anti-bot response, empty dynamic shell, or blocked post-list API. Diagnostics: ' + diagnostics.join(' | '),\n hashtags: '',\n thumbnail_url: '',\n views: '',\n likes: '',\n comments: '',\n shares: '',\n bookmarks: '',\n post_date: '',\n video_duration: ''\n }];\n }\n\n arr.forEach(item => {\n const row = document.createElement('div');\n row.className = 'uscraper-tiktok-row';\n const data = {\n row_type: item.row_type || 'video',\n video_url: item.video_url || '',\n video_id: item.video_id || '',\n creator: item.creator || uniqueId,\n profile_url: item.profile_url || (location.origin + '/@' + uniqueId),\n title_or_caption: item.title_or_caption || '',\n hashtags: item.hashtags || '',\n thumbnail_url: item.thumbnail_url || '',\n views: item.views || '',\n likes: item.likes || '',\n comments: item.comments || '',\n shares: item.shares || '',\n bookmarks: item.bookmarks || '',\n post_date: item.post_date || '',\n video_duration: item.video_duration || '',\n profile_following: profileFollowing,\n profile_followers: profileFollowers,\n profile_likes: profileLikes,\n page_url: location.href,\n page_title: document.title\n };\n Object.entries(data).forEach(([k, v]) => row.setAttribute('data-' + k.replace(/_/g, '-'), String(v ?? '')));\n row.textContent = data.title_or_caption || data.video_url || data.row_type;\n wrap.appendChild(row);\n });\n document.body.appendChild(wrap);\n return arr.length;\n }\n\n (async () => {\n try {\n dismiss();\n await scrollPass();\n addFromDom();\n scanScriptsAndHtml();\n if (items.size === 0) await fetchApiFallback();\n const rowsBuilt = buildRows();\n resolve({ rowsBuilt, videosFound: items.size, diagnostics, profile: uniqueId, pageTitle: document.title, pageUrl: location.href });\n } catch (e) {\n diagnostics.push('fatal=' + e.message);\n buildRows();\n resolve({ rowsBuilt: 1, videosFound: items.size, diagnostics, fatal: e.message });\n }\n })();\n});", "waitForCompletion": true, "timeout": 170 } }, { "block_id": "wait-for-element-2", "block_type": "process", "title": "Wait for Element", "description": "Wait until element appears", "position_x": 1920, "position_y": 260, "config": { "selector": "#uscraper-tiktok-rows .uscraper-tiktok-row", "timeout": 20, "visible": false } }, { "block_id": "structured-export-1", "block_type": "process", "title": "Structured Export", "description": "Export data with custom columns", "position_x": 2280, "position_y": 260, "config": { "rowSelector": "#uscraper-tiktok-rows .uscraper-tiktok-row", "fileName": "tiktok-scraper.csv", "saveLocation": "C:\\Users\\theskd\\Documents\\UScraper\\templates", "includeHeaders": true, "fileMode": "create", "columns": [ { "name": "row_type", "selector": "", "attribute": "data-row-type" }, { "name": "video_url", "selector": "", "attribute": "data-video-url" }, { "name": "video_id", "selector": "", "attribute": "data-video-id" }, { "name": "creator", "selector": "", "attribute": "data-creator" }, { "name": "profile_url", "selector": "", "attribute": "data-profile-url" }, { "name": "title_or_caption", "selector": "", "attribute": "data-title-or-caption" }, { "name": "hashtags", "selector": "", "attribute": "data-hashtags" }, { "name": "thumbnail_url", "selector": "", "attribute": "data-thumbnail-url" }, { "name": "views", "selector": "", "attribute": "data-views" }, { "name": "likes", "selector": "", "attribute": "data-likes" }, { "name": "comments", "selector": "", "attribute": "data-comments" }, { "name": "shares", "selector": "", "attribute": "data-shares" }, { "name": "bookmarks", "selector": "", "attribute": "data-bookmarks" }, { "name": "post_date", "selector": "", "attribute": "data-post-date" }, { "name": "video_duration", "selector": "", "attribute": "data-video-duration" }, { "name": "profile_following", "selector": "", "attribute": "data-profile-following" }, { "name": "profile_followers", "selector": "", "attribute": "data-profile-followers" }, { "name": "profile_likes", "selector": "", "attribute": "data-profile-likes" }, { "name": "page_url", "selector": "", "attribute": "data-page-url" }, { "name": "page_title", "selector": "", "attribute": "data-page-title" } ] } }, { "block_id": "end-1", "block_type": "output", "title": "End", "description": "Terminate execution flow", "position_x": 2640, "position_y": 260, "config": {} } ], "connections": [ { "from_block_id": "navigate-1", "from_connector_id": "right", "to_block_id": "wait-for-page-load-1", "to_connector_id": "left" }, { "from_block_id": "wait-for-page-load-1", "from_connector_id": "right", "to_block_id": "inject-javascript-1", "to_connector_id": "left" }, { "from_block_id": "inject-javascript-1", "from_connector_id": "right", "to_block_id": "wait-for-element-1", "to_connector_id": "left" }, { "from_block_id": "wait-for-element-1", "from_connector_id": "right", "to_block_id": "inject-javascript-2", "to_connector_id": "left" }, { "from_block_id": "inject-javascript-2", "from_connector_id": "right", "to_block_id": "wait-for-element-2", "to_connector_id": "left" }, { "from_block_id": "wait-for-element-2", "from_connector_id": "right", "to_block_id": "structured-export-1", "to_connector_id": "left" }, { "from_block_id": "structured-export-1", "from_connector_id": "right", "to_block_id": "end-1", "to_connector_id": "left" } ], "canvas_elements": [ { "id": "group-load", "element_type": "group", "title": "Page Load", "color": "#08bdba", "position_x": 48, "position_y": 156, "width": 2120, "height": 296, "z_index": 20, "data": { "memberBlockIds": [ "navigate-1", "wait-for-page-load-1", "wait-for-element-1", "wait-for-element-2" ] } }, { "id": "group-interaction", "element_type": "group", "title": "Interaction", "color": "#a56eff", "position_x": 768, "position_y": 156, "width": 1040, "height": 296, "z_index": 20, "data": { "memberBlockIds": [ "inject-javascript-1", "inject-javascript-2" ] } }, { "id": "group-extract", "element_type": "group", "title": "Data Extraction", "color": "#42be65", "position_x": 2208, "position_y": 156, "width": 380, "height": 296, "z_index": 20, "data": { "memberBlockIds": [ "structured-export-1" ] } }, { "id": "group-control", "element_type": "group", "title": "Control Flow", "color": "#8d8d8d", "position_x": 2568, "position_y": 156, "width": 380, "height": 296, "z_index": 20, "data": { "memberBlockIds": [ "end-1" ] } }, { "id": "note-overview", "element_type": "note", "title": "Overview", "content": "Best-effort TikTok public profile/video scraper. Default URL targets https://www.tiktok.com/@zachking because TikTok's official @tiktok profile returned profile stats but zero accessible post rows in the automated test session. Edit the Navigate URL for another public TikTok profile. The workflow dismisses popups, waits for the page, scrolls for dynamic/infinite-scroll content, scans rendered DOM and embedded JSON, then uses TikTok same-origin web API fallback endpoints to collect paginated post/video metadata. Exports video URL, creator, caption/title, hashtags, thumbnail, views, likes, comments, shares, bookmarks, post date, duration, and profile-level following/followers/likes. TikTok may still block automation with login/CAPTCHA/region/anti-bot checks; if no video data is accessible, a diagnostic row is exported.", "color": "#f1c21b", "position_x": 80, "position_y": 20, "width": 480, "height": 160, "z_index": 22, "data": {} }, { "id": "note-block-inject-javascript-1", "element_type": "note", "title": "Note: Inject JavaScript", "content": "Runs custom JavaScript in the page: `Array.from(document.querySelectorAll('button, [role=\"button\"]')).find(el => /^(accept|accept all|all...` Verify in browser if results are empty.", "color": "#ee5396", "position_x": 1040, "position_y": 240, "width": 340, "height": 140, "z_index": 22, "data": { "block_id": "inject-javascript-1" } }, { "id": "note-block-inject-javascript-2", "element_type": "note", "title": "Note: Inject JavaScript", "content": "Runs custom JavaScript in the page: `return new Promise(resolve => {\n const sleep = ms => new Promise(r => setTimeout(r, ms));\n const t...` Verify in browser if results are empty.", "color": "#ee5396", "position_x": 1760, "position_y": 240, "width": 340, "height": 140, "z_index": 22, "data": { "block_id": "inject-javascript-2" } }, { "id": "note-block-structured-export-1", "element_type": "note", "title": "Note: Structured Export", "content": "Extracts rows matching `#uscraper-tiktok-rows .uscraper-tiktok-row`. Confirm row count > 0 before running at scale.", "color": "#ee5396", "position_x": 2480, "position_y": 240, "width": 340, "height": 118, "z_index": 22, "data": { "block_id": "structured-export-1" } } ] }