U @6^@s0ddlZddlmZmZmZGdddeZdS)N) SpiderFootSpiderFootPluginSpiderFootEventc@sJeZdZdZddiZddiZdZefddZdd Z d d Z d d Z dS) sfp_crossrefz|Cross-Reference:Footprint:Crawling and Scanning::Identify whether other domains are associated ('Affiliates') of the target. checkbaseTzMCheck the base URL of the potential affiliate if no direct affiliation found?NcCs4||_||_t|D]}|||j|<qdS)N)sf tempStoragefetchedlistkeysopts)selfsfcuserOptsoptr7/var/www/spiderfoot.crq.systems/modules/sfp_crossref.pysetup$s zsfp_crossref.setupcCs ddddgS)NLINKED_URL_EXTERNAL SIMILARDOMAINCO_HOSTED_SITEZDARKNET_MENTION_URLrr rrr watchedEvents,szsfp_crossref.watchedEventscCsddgS)NAFFILIATE_INTERNET_NAMEAFFILIATE_WEB_CONTENTrrrrrproducedEvents2szsfp_crossref.producedEventsc Csv|j}|j}|j}|jd|d||dkr>d|}||j|rl|jd|ddS||j kr|jd|ddSd|j |<|jd ||jj ||j d |j d d d }|ddkr|jd|ddSd}| D]D}t d|dt j}t ||d} t| dkrd}|} qBq|s|dkr|j dr|j|} | |j krzdSd|j | <|jj | |j d |j d d d }|ddk r| D]>}t d|dt j}t ||d} t| dkrd}q|rr|jsd|_|jd| td|j| |j|} |j| _|| td|d|j| } |j| _|| dS)NzReceived event, z, from )rrzhttp://z Ignoring z as not externalz as already testedTzTesting for affiliation: _fetchtimeout _useragenti)timeout useragent sizeLimitcontentz as no data returnedFz ([\.'\/"\ ]z [\.'\/"\ ])rrrz ['\/"\ ])UnknownzFound affiliate: rr) eventTypemoduledatardebuglower getTargetmatchesurlFQDNr fetchUrlr getNamesrecompile IGNORECASEfindalllen urlBaseUrlmoduleDataSourceinfor__name__notifyListeners) r event eventName srcModuleName eventDataresmatchednamepatr)urlZevt1Zevt2rrr handleEvent8s~          zsfp_crossref.handleEvent) r5 __module__ __qualname____doc__r optdescsr dictrrrr@rrrrrsr)r-sflibrrrrrrrrs