3 @6^@s0ddlZddlmZmZmZGdddeZdS)N) SpiderFootSpiderFootPluginSpiderFootEventc@sJeZdZdZddiZddiZdZefddZdd Z d d Z d d Z dS) sfp_crossrefz|Cross-Reference:Footprint:Crawling and Scanning::Identify whether other domains are associated ('Affiliates') of the target. checkbaseTzMCheck the base URL of the potential affiliate if no direct affiliation found?NcCs8||_|j|_x"t|jD]}|||j|<qWdS)N)sf tempStoragefetchedlistkeysopts)selfsfcuserOptsoptr7/var/www/spiderfoot.crq.systems/modules/sfp_crossref.pysetup$s zsfp_crossref.setupcCs ddddgS)NLINKED_URL_EXTERNAL SIMILARDOMAINCO_HOSTED_SITEZDARKNET_MENTION_URLr)r rrr watchedEvents,szsfp_crossref.watchedEventscCsddgS)NAFFILIATE_INTERNET_NAMEAFFILIATE_WEB_CONTENTr)r rrrproducedEvents2szsfp_crossref.producedEventsc Cs~|j}|j}|j}|jjd|d||dkr>d|j}|jj|jj|rl|jjd|ddS||j kr|jjd|ddSd |j |<|jjd ||jj ||j d |j d d d}|ddkr|jjd|ddSd}xT|jj D]D}t jd|dt j}t j||d} t| dkrd }|} PqW|s|dkr|j dr|jj|} | |j kr~dSd |j | <|jj | |j d |j d d d}|ddk rxN|jj D]>}t jd|dt j}t j||d} t| dkrd }qW|rz|jsd|_|jjd| td|jj| |j|} |j| _|j| td|d|j| } |j| _|j| dS)NzReceived event, z, from rrzhttp://z Ignoring z as not externalz as already testedTzTesting for affiliation: _fetchtimeout _useragenti)timeout useragent sizeLimitcontentz as no data returnedFz ([\.'\/"\ ]z [\.'\/"\ ])rrrz ['\/"\ ])UnknownzFound affiliate: rr)rr) eventTypemoduledatardebuglower getTargetmatchesurlFQDNr fetchUrlr getNamesrecompile IGNORECASEfindalllen urlBaseUrlmoduleDataSourceinfor__name__notifyListeners) r event eventName srcModuleName eventDataresmatchednamepatr(urlZevt1Zevt2rrr handleEvent8sn            zsfp_crossref.handleEvent) r4 __module__ __qualname____doc__r optdescsr dictrrrr?rrrrrsr)r,sflibrrrrrrrrs