Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reddit.com: privacy #18938

Open
8 of 9 tasks
partingscientist opened this issue Jul 11, 2023 · 39 comments
Open
8 of 9 tasks

reddit.com: privacy #18938

partingscientist opened this issue Jul 11, 2023 · 39 comments
Labels
ongoing ongoing issue

Comments

@partingscientist
Copy link
Contributor

partingscientist commented Jul 11, 2023

Prerequisites

  • I read and understand the policy about what is a valid filter issue.
  • I verified that this issue is not a duplicate. (Use this button to find out.)
  • I forced an update of my filter lists. (Click the "Purge all caches" button while holding the 'Shift' key, then click the "Update now" button.)
  • I did not remove any of the default filter lists, or I have verified that the issue was not caused by removing any of the default lists.
  • I did not enable external filter lists, or I have verified that the issue still occurs without enabling external filter lists.
  • I do not have custom filters/rules, or I have verified that the issue still occurs without custom filters/rules.
  • I am not using uBlock Origin (uBO) along with other content blocker extensions.
  • I have verified that the web browser's built-in blocker or DNS blocking (standalone or through a VPN) is not causing the issue.
  • I did not answer truthfully to ALL the above checkpoints.

URL address of the web page

https://www.reddit.com/r/worldnews/

Category

privacy

Description

While not logged in, Reddit mobile site tracks outbound navigation using out.reddit.com.
The behaviour still exists after testing with Firefox 116.0b3 and default uBlock Origin.

Other extensions used

None.

Screenshot(s)

Screenshot(s)

Configuration

uBlock Origin 1.50.0:
 since: 27d.9h.6m
Firefox Mobile: 113
filterset (summary):
 network: 127068
 cosmetic: 196662
 scriptlet: 27009
 html: 1171
listset (total-discarded, last-updated):
 added:
  https://gitlab.com/magnolia1234/bypass-paywalls-clean-filters/-/raw/main/bpc-paywall-filter.txt: 961-0, now
  https://raw.githubusercontent.com/DandelionSprout/adfilt/master/LegitimateURLShortener.txt: 2230-0, now
  adguard-spyware-url: 1117-147, 1m
  block-lan: 47-0, now
  curben-pup: 189-0, now
  adguard-social: 20887-822, now
  fanboy-cookiemonster: 47637-3839, now
  fanboy-social: 16098-8, now
  ublock-annoyances: 5344-16, now
  dpollock-0: 11524-468, now
  IDN-0: 9527-4, now
  adguard-cookies: 26293-61, now
  [8 lists not shown]: [too many]
 default:
  user-filters: 25-0, never
  ublock-filters: 35059-525, 1m
  ublock-badware: 7532-1, 1m
  ublock-privacy: 469-2, 1m
  ublock-unbreak: 2117-1, 1m
  ublock-quick-fixes: 350-0, 1h.58m
  adguard-mobile: 8383-141, 1m
  easylist: 68120-1060, 1m
  easyprivacy: 33422-906, now
  urlhaus-1: 6440-1, now
  plowe-0: 3740-1, now
filterset (user): [array of 25 redacted]
hostRuleset:
 added: [array of 2 redacted]
userSettings:
 advancedUserEnabled: true
 colorBlindFriendly: true
 ignoreGenericCosmeticFilters: false
 webrtcIPAddressHidden: true
hiddenSettings:
 userResourcesLocation: [redacted]
supportStats:
 allReadyAfter: 2482 ms
 maxAssetCacheWait: 1543 ms
popupPanel:
 blocked: 10
 network:
  reddit.com: 1
  amazon-adsystem.com: 1
  googletagservices.com: 1
  redditmedia.com: 5
  redditstatic.com: 2
 extended:
  ##.adsense-ads
  ##.googads
  ##.hasads
  ##.native-ad
  ##.HeaderAd
  ##.NavBarAd
  ##.ad-medium
  ##.post-ad
  ##.promoad
  ##.rectad
  ##.sidebar-ad:not(.adsbygoogle)
  ##.small-ad
  ##.sponsorAd
  ##.sponsorad
  ##.sponsorPost
  ##.PostFooter__share
  ##.XPromoPopupRpl
  ##article[id] > a[href^="https://alb.reddit.com/"] ~ *
  ##.XPromoPopupRpl.m-active
  ##.promotedlink
  ##.PostsList > div article:has(> .Post__header-wrapper .PostHead…
  ##.PostsList > div:not([class]):not([id]) > div:not([class]):not…
  ##.size-compact.Post:has([class*="promoted"])
  ##article.Post:has(a[rel="nofollow noopener sponsored"])
  ##body:style(pointer-events:auto!important;)
  ##body:style(overflow: auto !important;)
  ##body.scroll-disabled:style(overflow: visible!important; positi…
  ##body,html.scroll-is-blocked:style(overflow: auto !important;)
  ##body:style(pointer-events:auto !important;)
  ##body:style(pointer-events: auto !important; overflow: auto !im…
  ##body.scroll-disabled:style(overflow: visible !important; posit…
  ###acceptabletest:style(display: block !important;)
  ###adblocktest:style(display: block !important;)
  ##+js(no-setTimeout-if, /^/)
  ##+js(no-setInterval-if, /^/)
  ##+js(addEventListener-defuser, touchmove)
  ##+js(set-constant, Object.prototype.allowClickTracking, false)
  ##+js(no-fetch-if, url:/^https:\/\/www\.reddit\.com$/ method:pos…
  ##+js(no-xhr-if, method:POST url:/^https:\/\/www\.reddit\.com$/)
  ##+js(replace-node-text, script, "outboundUrl", "outbound")
  ##+js(json-prune, data.*.elements.edges.[].node.outboundLink)
  ##+js(addEventListener-defuser, block-body-scrolling)
  ##+js(remove-class, m-blurred, .m-blurred)
  ##+js(remove-class, scroll-disabled, body)
  ##+js(set-constant, Notification.requestPermission, noopFunc)
@stephenhawk8054
Copy link
Member

Can you test this?

reddit.com,~out.reddit.com##+js(set, Object.prototype.allowClickTracking, false)

@partingscientist
Copy link
Contributor Author

partingscientist commented Jul 11, 2023

I have that on my configuration (as attached above) when testing on my dirty profile and the behaviour still persists. I'll test again to make sure.

Edit: Yeah, still persists

@stephenhawk8054
Copy link
Member

Does it occur on desktop?

@partingscientist
Copy link
Contributor Author

I cannot reproduce on desktop using clean profile and default uBlock Origin, only on mobile.

@stephenhawk8054 stephenhawk8054 added the ongoing ongoing issue label Jul 13, 2023
@uBlockOrigin uBlockOrigin deleted a comment Jul 13, 2023
@mapx-
Copy link
Contributor

mapx- commented Jul 13, 2023

@partingscientist are you able to reproduce the issue using FF emulator on desktop ?

How did you note that behaviour on android ? steps ?

@partingscientist
Copy link
Contributor Author

partingscientist commented Jul 13, 2023

Are you able to reproduce the issue using FF emulator on desktop ?

Yes, on clean profile and default uBlock Origin.

How did you note that behaviour on android ? steps ?

Given https://www.reddit.com/r/worldnews/ as a subreddit example, I just open one of the article links posted while having the logger being opened.

@mapx-
Copy link
Contributor

mapx- commented Jul 13, 2023

I cannot reproduce the issue => no out.reddit.com in the logger (even disabling the scriptlets on reddit)

@gorhill
Copy link
Member

gorhill commented Jul 13, 2023

I also cannot reproduce on mobile.

@mapx- mapx- added unable to reproduce and removed ongoing ongoing issue labels Jul 13, 2023
@partingscientist
Copy link
Contributor Author

partingscientist commented Jul 14, 2023

I'm still investigating the reproducibility on my end, as I can confidently reproduce the issue; I'll report my findings soon. Just to make sure, you're not logged in while testing, right?

While not logged in, Reddit mobile site tracks outbound navigation using out.reddit.com.

@partingscientist
Copy link
Contributor Author

Here's what I found (Firefox Mobile 116.0b3 and default stable-release uBlock Origin on Android):

It seems to me that Reddit is currently testing a new user interface for their mobile website (reddit.com).

Here's the old user interface

old

Here's what they are currently testing

new

The issue is reproducible on the old interface by interacting with the link bar below the post thumbnail on the right hand side. However, the issue is not reproducible on the new interface. I very rarely get this new interface, perhaps due to geolocation and other factors which also might vary amongst the maintainers here.

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Jul 14, 2023

I can reproduce. The key is this

interacting with the link bar below the post thumbnail on the right hand side

Can you test this filter?

reddit.com##+js(json-prune, data.children.[].data.outbound_link)

An easy way to test if there's out.reddit.com link is adding the filter ||out.reddit.com^.

@partingscientist
Copy link
Contributor Author

reddit.com##+js(json-prune, data.children.[].data.outbound_link)

This works, both on Firefox Mobile and Firefox Desktop with the emulator.

@uBlock-user
Copy link
Contributor

uBlock-user commented Aug 8, 2023

Tracking happening via urls like https://www.reddit.com/svc/shreddit/b52ae76b b52ae76b is randomised

@stephenhawk8054

filter -- /^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z0-9]{8}$/$xhr,1p,method=post

@uBlock-user
Copy link
Contributor

@stephenhawk8054 I'm not sure if these two are needed -

reddit.com##+js(no-fetch-if, url:/^https:\/\/www\.reddit\.com$/ method:post)
reddit.com##+js(no-xhr-if, method:POST url:/^https:\/\/www\.reddit\.com$/)

I don't have a reddit account, so cannot confirm it but you can.

@stephenhawk8054
Copy link
Member

You're right. I don't see those connections any more.

stephenhawk8054 added a commit that referenced this issue Aug 9, 2023
@partingscientist
Copy link
Contributor Author

partingscientist commented Aug 25, 2023

Tracking happening via urls like https://www.reddit.com/svc/shreddit/b52ae76b b52ae76b is randomised

filter -- /^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z0-9]{8}$/$xhr,1p,method=post

@stephenhawk8054 the length of pathname characters is randomised.

Source

https://www.redditstatic.com/shreddit/en-US/shell-594c01eb.js, line 4477

// Ho() returns a UUID in the form [0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12} 
// (line 2665-2677, line 2734-2742)
const us = `/svc/shreddit/${(()=>{const e=Ho().split("-"),t=1+Math.floor(6*Math.random()),n=`${e[1]+e[2]}`.substring(0,t);return e[0]+n})()}`,
   ms = (e, t) => {
      const n = "function" != typeof navigator.sendBeacon || ps() ? "function" == typeof fetch ? cs : void 0 : ds;
      void 0 !== n && n({
         url: us,
         eventsBuffer: e,
         microapp: _r(),
         shouldNotSendDropEvent: t
      })
   };

/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z0-9]{8,14}$/$xhr,1p,method=post,match-case should work, I think.

@partingscientist
Copy link
Contributor Author

It cannot be 8 characters actually (my bad), e[0] is always 8 characters and n is 1-6 characters. I'm assuming you want to keep the 8 in case of partial rollout of the new one. Just letting you know to make sure.

@stephenhawk8054
Copy link
Member

Keeping 8 is still good as the old ones have 8: https://www.reddit.com/svc/shreddit/b52ae76b. This URL hardly has FPs anyway.

@uBlock-user
Copy link
Contributor

Tracking happening via urls like https://www.reddit.com/svc/shreddit/b52ae76b b52ae76b is randomised
filter -- /^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z0-9]{8}$/$xhr,1p,method=post

@stephenhawk8054 the length of pathname characters is randomised.
Source

/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z0-9]{8,14}$/$xhr,1p,method=post,match-case should work, I think.

When I suggested that filter it was always 8 during my tests. So did something change now ?

@uBlock-user
Copy link
Contributor

@stephenhawk8054 Why change a-z to a-f ?

@stephenhawk8054
Copy link
Member

I did see a shreddit connection with 10 characters few hours ago, and there's actually a https://www.redditstatic.com/shreddit/en-US/shell- script and inside it does contain the code in #18938 (comment)

var Co = /^(?:[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}|00000000-0000-0000-0000-000000000000)$/i;

function Ao(e) {
   return "string" == typeof e && Co.test(e)
}

function Oo(e) {
   var t = arguments.length > 1 && void 0 !== arguments[1] ? arguments[1] : 0,
      n = (Lo[e[t + 0]] + Lo[e[t + 1]] + Lo[e[t + 2]] + Lo[e[t + 3]] + "-" + Lo[e[t + 4]] + Lo[e[t + 5]] + "-" + Lo[e[t + 6]] + Lo[e[t + 7]] + "-" + Lo[e[t + 8]] + Lo[e[t + 9]] + "-" + Lo[e[t + 10]] + Lo[e[t + 11]] + Lo[e[t + 12]] + Lo[e[t + 13]] + Lo[e[t + 14]] + Lo[e[t + 15]]).toLowerCase();
   if (!Ao(n)) throw TypeError("Stringified UUID is invalid");
   return n
}

function Ho(e, t, n) {
   var o = (e = e || {}).random || (e.rng || Io)();
   if (o[6] = 15 & o[6] | 64, o[8] = 63 & o[8] | 128, t) {
      n = n || 0;
      for (var r = 0; r < 16; ++r) t[n + r] = o[r];
      return t
   }
   return Oo(o)
}

const ps = () => ("boolean" == typeof hs || (e => {
      hs = e
   })(document.querySelector("shreddit-app")?.hasAttribute("disable-send-beacon") ?? !1), hs),
   us = `/svc/shreddit/${(()=>{const e=Ho().split("-"),t=1+Math.floor(6*Math.random()),n=`${e[1]+e[2]}`.substring(0,t);return e[0]+n})()}`,
   ms = (e, t) => {
      const n = "function" != typeof navigator.sendBeacon || ps() ? "function" == typeof fetch ? cs : void 0 : ds;
      void 0 !== n && n({
         url: us,
         eventsBuffer: e,
         microapp: _r(),
         shouldNotSendDropEvent: t
      })
   };

The characters indeed only had [a-f], but I forgot to save the scripts and connections there. And now I cannot see those shreddit connections any more. Does anyone see any connections slipped through?

@stephenhawk8054
Copy link
Member

Update: Ah ok I can see it now: https://www.reddit.com/svc/shreddit/c8cea5df7d8b4

image

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Aug 25, 2023

I also see the query parameter rdt in reddit link, should we add it to our query stripping section?


Might be related: https://reddit.my.site.com/helpcenter/s/article/Reddit-Ads-Pixel-Retargeting

@uBlock-user
Copy link
Contributor

I think you should revert to a-z, they can change it anytime, so a-z would last long.

stephenhawk8054 added a commit that referenced this issue Aug 25, 2023
@stephenhawk8054
Copy link
Member

Ok: eee6fec

@gorhill
Copy link
Member

gorhill commented Aug 25, 2023

they can change it anytime

Usually we try to be as specific as needed to avoid false positives. Using a-f was the proper approach if we are dealing with hex-based names. If we craft according to "they can change it anytime" logic, then why even mind what comes after .../shreddit/.

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Aug 25, 2023

Hmmm... the hex-based is what I really didn't think about, I was always wondering why reddit used [0-9a-f] in their script.

@uBlock-user
Copy link
Contributor

uBlock-user commented Aug 25, 2023

If we craft according to "they can change it anytime" logic, then why even mind what comes after .../shreddit/.

So what was I suggesting was not the proper approach ? My point is this wouldn't last long if we go from a-z to a-f and I want it to last long and not make it easier for reddit in that place.

then why even mind what comes after .../shreddit/.

Because I'm not that paranoid. Also to mention, the previous filters I suggested lasted more than one and a half years(Oct 2021 - August 2023), so I'm going by that way.

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Aug 26, 2023

I think it's ok, reddit does indeed usually change that part. Actually they have changed it to [A-Za-z] and longer path already.


The blocked requests just keeps increasing so I want to try no-fetch/xhr-if but these don't work?

reddit.com##+js(no-fetch-if, 'url:/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[0-9A-z]{9,}$/ method:post')
reddit.com##+js(no-xhr-if, 'url:/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[0-9A-z]{9,}$/ method:post')

stephenhawk8054 added a commit that referenced this issue Sep 3, 2023
@partingscientist
Copy link
Contributor Author

@stephenhawk8054 I'm not sure if these two are needed -

reddit.com##+js(no-fetch-if, url:/^https:\/\/www\.reddit\.com$/ method:post)
reddit.com##+js(no-xhr-if, method:POST url:/^https:\/\/www\.reddit\.com$/)

I don't have a reddit account, so cannot confirm it but you can.

I found a connection to that url, here is a test link: https://www.reddit.com/gallery/16dgs9k.

I tested these but they do not work.

reddit.com##+js(no-fetch-if, url:/^https:\/\/www\.reddit\.com\/$/ method:post)
reddit.com##+js(no-xhr-if, url:/^https:\/\/www\.reddit\.com\/$/ method:post)

@stephenhawk8054
Copy link
Member

@partingscientist Does this work?

||www.reddit.com/|$xhr,1p,method=post

@uBlock-user
Copy link
Contributor

uBlock-user commented Sep 9, 2023

/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[a-z-0-9]{8,}$/$xhr,1p,method=post still works.

stephenhawk8054 added a commit that referenced this issue Sep 9, 2023
@partingscientist
Copy link
Contributor Author

@stephenhawk8054

// https://www.redditstatic.com/shreddit/en-US/shell-e5fddcc3.js, 6009
let Tc;
const Ic = () => ("boolean" == typeof Tc || (e => {
      Tc = e
   })(document.querySelector("shreddit-app")?.hasAttribute("disable-send-beacon") ?? !1), Tc),
   Cc = (e, t) => Math.floor(Math.random() * (t - e) + e),
   Ac = (e, t, n) => `${e.slice(0,n)}${t}${e.slice(n)}`,
   Lc = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789",
   Pc = () => Lc[Math.floor(Math.random() * Lc.length)],
   Oc = `/svc/shreddit/${(e=>{if(e<="/!".length)throw new Error(`Invalid len argument supplied. Must be greater than ${"/!".length}`);let t=[...Array(e-"/!".length)].map(Pc).join("");return t=Ac(t,"/!"[0],Math.floor(t.length/2)),t=Ac(t,"/!"[1],Cc(0,t.length-1)),t})(Cc(15,25))}`,
   Rc = ({
      eventsBuffer: e,
      v2EventsRoute: t,
      options: n
   }) => {
      const o = "function" != typeof navigator.sendBeacon || Ic() ? "function" == typeof fetch ? Sc : void 0 : kc;
      void 0 !== o && o({
         url: t || Oc,
         eventsBuffer: e,
         microapp: ar(),
         options: n
      })
   };
# https://github.com/uBlockOrigin/uAssets/blame/2789c0c31a3ee62e432e3f5090398553075d5f74/filters/privacy.txt#L109-L110
||www.reddit.com/svc/shreddit/$xhr,1p,method=post
@@||reddit.com/svc/shreddit/more-comments/$xhr,1p,method=post

?

@stephenhawk8054
Copy link
Member

@partingscientist Looks like the above filters cannot load the comments in this link? https://www.reddit.com/r/meirl/comments/17nyl7a/meirl/

@partingscientist
Copy link
Contributor Author

I thought I'd let you know first, haven't attempted to craft or test anything previously.

How about

/^https:\/\/www\.reddit\.com\/svc\/shreddit\/[A-z0-9\/!]{15,24}$/$xhr,1p,method=post,match-case

?

@stephenhawk8054
Copy link
Member

@partingscientist Still does not work for me. If I'm not mistaken, it's because reddit is detecting if you block those XHR, it won't load the comments.

Even when using no-fetch-if, it still has issue due to this

fd = async ({
  url: e,
  eventsBuffer: t,
  microapp: n,
  options: o
}) => {
  const r = jo.get("csrf_token") ?? "",
     i = !yd() ? t : {
        csrf_token: r,
        info: t
     },
     a = JSON.stringify(i);
  let s;
  try {
     s = await fetch(e, {
        body: a,
        headers: {
           "Content-Type": "text/PLAIN",
           "x-sh-microapp-route": n || "monolith"
        },
        keepalive: !0,
        method: "post",
        credentials: "same-origin"
     })
  } catch (n) {
     return void(o?.shouldNotRetryIfFail ? o?.shouldNotSendDropEvent || bd({
        eventsBuffer: t,
        errorText: `An error: ${n}`,
        v2EventsRoute: e
     }) : Id({
        eventsBuffer: t,
        v2EventsRoute: e,
        options: {
           shouldNotRetryIfFail: !0
        }
     }))
  }
  if (s?.ok) {
     Array.from(s.headers).length && (vd || (vd = document.querySelector("shreddit-app")), vd)?.resolvePendingRequests();
     const e = s.headers.get("x-set-loid") || void 0,
        t = s.headers.get("x-set-session") || void 0;
     window.dispatchEvent(new CustomEvent("v2-events-sent", {
        detail: {
           loid: e,
           session_tracker: t
        }
     }))
  } else o?.shouldNotSendDropEvent || bd({
     eventsBuffer: t,
     errorText: `HTTP Response Code: ${s?.status}`,
     v2EventsRoute: e
  });
  return s
};

(This is their old code, they might change something already)

@uBlock-user
Copy link
Contributor

it's because reddit is detecting if you block those XHR, it won't load the comments.

Technically, it's checking for specific request headers with the fetch request, if the headers are not found, content won't load.

@uBlock-user
Copy link
Contributor

There's two ways to this tracking issue as I can see -

  • Intercept the request and delete the payload before passing the request, so when the request reaches the server, there's no data, so no tracking(All the scriptlets that gorhill has added that intercept are so far only focusing on response)

  • Mimic the request headers when using no-fetch-if and hence fulfill the requirements for the content to load.

@stephenhawk8054

@uBlock-user uBlock-user reopened this Jun 23, 2024
@uBlock-user uBlock-user added the ongoing ongoing issue label Jun 23, 2024
@uBlock-user
Copy link
Contributor

@gorhill any solution for ongoing issue with reddit ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ongoing ongoing issue
Projects
None yet
Development

No branches or pull requests

5 participants