Javascript：正则表达式将所有相对 URL 更改为绝对

Javascript: REGEX to change all relative Urls to Absolute

提问by Trevor

I'm currently creating a Node.js webscraper/proxy, but I'm having trouble parsing relative Urls found in the scripting part of the source, I figured REGEX would do the trick. Although it is unknown how I would achieve this.

我目前正在创建一个 Node.js 网络爬虫/代理，但我在解析源代码脚本部分中的相关 URL 时遇到问题，我认为 REGEX 可以解决问题。虽然不知道我将如何实现这一目标。

Is there anyway I can go about this?

无论如何我可以解决这个问题吗？

Also I'm open to an easier way of doing this, as I'm quite baffle about how other proxies parse websites. I figured that most are just glorified site scrapers that can read a site's source a relay all links/forms back to the proxy.

此外，我愿意采用更简单的方法来做到这一点，因为我对其他代理如何解析网站感到非常困惑。我认为大多数只是美化的站点抓取工具，可以读取站点的源并将所有链接/表单中继回代理。

回答by Rob W

Advanced HTML string replacement functions

高级 HTML 字符串替换功能

Note for OP, because he requested such a function: Change base_urlto your proxy's basE URL in order to achieve the desired results.

请注意 OP，因为他请求了这样一个功能：更改base_url为您的代理的基本 URL 以达到预期的结果。

Two functions will be shown below (the usage guide is contained within the code). Make sure that you don't skip any part of the explanation of this answer to fully understand the function's behaviour.

下面将展示两个函数（使用指南包含在代码中）。确保您没有跳过此答案解释的任何部分以完全理解该函数的行为。

• rel_to_abs(urL)- This function returns absolute URLs. When an absolute URL with a commonly trusted protocol is passed, it will immediately return this URL. Otherwise, an absolute URL is generated from the base_urland the function argument. Relative URLs are correctly parsed (../; ./; .; //).
• replace_all_rel_by_abs- This function will parse alloccurences of URLs which have a significant meaning in HTML, such as CSS url(), links and external resources. See the code for a full list of parsed instances. See this answerfor an adjusted implementation to sanitise HTML stringsfrom an external source (to embed in the document).
• Test case (at the bottom of the answer): To test the effectiveness of the function, simply paste the bookmarklet at the location's bar.

• rel_to_abs(urL)- 此函数返回绝对 URL。当一个具有普遍信任协议的绝对 URL 被传递时，它会立即返回这个 URL。否则，将根据base_url和 函数参数生成绝对 URL 。正确解析相对 URL ( ../; ./; .; //)。
• replace_all_rel_by_abs- 此函数将解析所有出现在 HTML 中具有重要意义的 URL，例如 CSS url()、链接和外部资源。有关已解析实例的完整列表，请参阅代码。请参阅此答案以了解从外部源（以嵌入到文档中）来清理 HTML 字符串的调整实现。
• 测试用例（在答案底部）：要测试该功能的有效性，只需将书签粘贴到该位置的栏中。

---

---

rel_to_abs- Parsing relative URLsrel_to_abs-解析相对 URL

function rel_to_abs(url){
    /* Only accept commonly trusted protocols:
     * Only data-image URLs are accepted, Exotic flavours (escaped slash,
     * html-entitied characters) are not supported to keep the function fast */
  if(/^(https?|file|ftps?|mailto|javascript|data:image\/[^;]{2,9};):/i.test(url))
         return url; //Url is already absolute

    var base_url = location.href.match(/^(.+)\/?(?:#.+)?$/)[0]+"/";
    if(url.substring(0,2) == "//")
        return location.protocol + url;
    else if(url.charAt(0) == "/")
        return location.protocol + "//" + location.host + url;
    else if(url.substring(0,2) == "./")
        url = "." + url;
    else if(/^\s*$/.test(url))
        return ""; //Empty = Return nothing
    else url = "../" + url;

    url = base_url + url;
    var i=0
    while(/\/\.\.\//.test(url = url.replace(/[^\/]+\/+\.\.\//g,"")));

    /* Escape certain characters to prevent XSS */
    url = url.replace(/\.$/,"").replace(/\/\./g,"").replace(/"/g,"%22")
            .replace(/'/g,"%27").replace(/</g,"%3C").replace(/>/g,"%3E");
    return url;
}

Cases / examples:

案例/例子：

• http://foo.bar. Already an absolute URL, thus returned immediately.
• /dooRelative to the root: Returns the current root + provided relative URL.
• ./mehRelative to the current directory.
• ../boohRelative to the parent directory.

• http://foo.bar. 已经是绝对URL，因此立即返回。
• /doo相对于根：返回当前根 + 提供的相对 URL。
• ./meh相对于当前目录。
• ../booh相对于父目录。

The function converts relative paths to ../, and performs a search-and-replace (http://domain/sub/anything-but-a-slash/../meto http://domain/sub/me).

该函数将相对路径转换为../，并执行搜索和替换 ( http://domain/sub/anything-but-a-slash/../meto http://domain/sub/me)。

---

---

replace_all_rel_by_abs- Convert all relevant occurences of URLsreplace_all_rel_by_abs-转换


脚本实例中所有相关的 URLsURLs ( <script><script>, 事件处理程序是not没有被替换，因为创建一个快速且安全的过滤器来解析 JavaScript 几乎是不可能的。

This script is served with some comments inside. Regular Expressions are dynamically created, because an individual RE can have a size of 3000characters. <meta http-equiv=refresh content=.. >can be obfuscated in various ways, hence the size of the RE.

这个脚本里面有一些注释。正则表达式是动态创建的，因为单个 RE 的大小可以为3000 个字符。<meta http-equiv=refresh content=.. >可以通过各种方式混淆，因此 RE 的大小。

function replace_all_rel_by_abs(html){
    /*HTML/XML Attribute may not be prefixed by these characters (common 
       attribute chars.  This list is not complete, but will be sufficient
       for this function (see http://www.w3.org/TR/REC-xml/#NT-NameChar). */
    var att = "[^-a-z0-9:._]";

    var entityEnd = "(?:;|(?!\d))";
    var ents = {" ":"(?:\s| ?|&#0*32"+entityEnd+"|&#x0*20"+entityEnd+")",
                "(":"(?:\(|&#0*40"+entityEnd+"|&#x0*28"+entityEnd+")",
                ")":"(?:\)|&#0*41"+entityEnd+"|&#x0*29"+entityEnd+")",
                ".":"(?:\.|&#0*46"+entityEnd+"|&#x0*2e"+entityEnd+")"};
                /* Placeholders to filter obfuscations */
    var charMap = {};
    var s = ents[" "]+"*"; //Short-hand for common use
    var any = "(?:[^>\"']*(?:\"[^\"]*\"|'[^']*'))*?[^>]*";
    /* ^ Important: Must be pre- and postfixed by < and >.
     *   This RE should match anything within a tag!  */

    /*
      @name ae
      @description  Converts a given string in a sequence of the original
                      input and the HTML entity
      @param String string  String to convert
      */
    function ae(string){
        var all_chars_lowercase = string.toLowerCase();
        if(ents[string]) return ents[string];
        var all_chars_uppercase = string.toUpperCase();
        var RE_res = "";
        for(var i=0; i<string.length; i++){
            var char_lowercase = all_chars_lowercase.charAt(i);
            if(charMap[char_lowercase]){
                RE_res += charMap[char_lowercase];
                continue;
            }
            var char_uppercase = all_chars_uppercase.charAt(i);
            var RE_sub = [char_lowercase];
            RE_sub.push("&#0*" + char_lowercase.charCodeAt(0) + entityEnd);
            RE_sub.push("&#x0*" + char_lowercase.charCodeAt(0).toString(16) + entityEnd);
            if(char_lowercase != char_uppercase){
                /* Note: RE ignorecase flag has already been activated */
                RE_sub.push("&#0*" + char_uppercase.charCodeAt(0) + entityEnd);   
                RE_sub.push("&#x0*" + char_uppercase.charCodeAt(0).toString(16) + entityEnd);
            }
            RE_sub = "(?:" + RE_sub.join("|") + ")";
            RE_res += (charMap[char_lowercase] = RE_sub);
        }
        return(ents[string] = RE_res);
    }

    /*
      @name by
      @description  2nd argument for replace().
      */
    function by(match, group1, group2, group3){
        /* Note that this function can also be used to remove links:
         * return group1 + "javascript://" + group3; */
        return group1 + rel_to_abs(group2) + group3;
    }
    /*
      @name by2
      @description  2nd argument for replace(). Parses relevant HTML entities
      */
    var slashRE = new RegExp(ae("/"), 'g');
    var dotRE = new RegExp(ae("."), 'g');
    function by2(match, group1, group2, group3){
        /*Note that this function can also be used to remove links:
         * return group1 + "javascript://" + group3; */
        group2 = group2.replace(slashRE, "/").replace(dotRE, ".");
        return group1 + rel_to_abs(group2) + group3;
    }
    /*
      @name cr
      @description            Selects a HTML element and performs a
                                search-and-replace on attributes
      @param String selector  HTML substring to match
      @param String attribute RegExp-escaped; HTML element attribute to match
      @param String marker    Optional RegExp-escaped; marks the prefix
      @param String delimiter Optional RegExp escaped; non-quote delimiters
      @param String end       Optional RegExp-escaped; forces the match to end
                              before an occurence of <end>
     */
    function cr(selector, attribute, marker, delimiter, end){
        if(typeof selector == "string") selector = new RegExp(selector, "gi");
        attribute = att + attribute;
        marker = typeof marker == "string" ? marker : "\s*=\s*";
        delimiter = typeof delimiter == "string" ? delimiter : "";
        end = typeof end == "string" ? "?)("+end : ")(";
        var re1 = new RegExp('('+attribute+marker+'")([^"'+delimiter+']+'+end+')', 'gi');
        var re2 = new RegExp("("+attribute+marker+"')([^'"+delimiter+"]+"+end+")", 'gi');
        var re3 = new RegExp('('+attribute+marker+')([^"\'][^\s>'+delimiter+']*'+end+')', 'gi');
        html = html.replace(selector, function(match){
            return match.replace(re1, by).replace(re2, by).replace(re3, by);
        });
    }
    /* 
      @name cri
      @description            Selects an attribute of a HTML element, and
                                performs a search-and-replace on certain values
      @param String selector  HTML element to match
      @param String attribute RegExp-escaped; HTML element attribute to match
      @param String front     RegExp-escaped; attribute value, prefix to match
      @param String flags     Optional RegExp flags, default "gi"
      @param String delimiter Optional RegExp-escaped; non-quote delimiters
      @param String end       Optional RegExp-escaped; forces the match to end
                                before an occurence of <end>
     */
    function cri(selector, attribute, front, flags, delimiter, end){
        if(typeof selector == "string") selector = new RegExp(selector, "gi");
        attribute = att + attribute;
        flags = typeof flags == "string" ? flags : "gi";
        var re1 = new RegExp('('+attribute+'\s*=\s*")([^"]*)', 'gi');
        var re2 = new RegExp("("+attribute+"\s*=\s*')([^']+)", 'gi');
        var at1 = new RegExp('('+front+')([^"]+)(")', flags);
        var at2 = new RegExp("("+front+")([^']+)(')", flags);
        if(typeof delimiter == "string"){
            end = typeof end == "string" ? end : "";
            var at3 = new RegExp("("+front+")([^\"'][^"+delimiter+"]*" + (end?"?)("+end+")":")()"), flags);
            var handleAttr = function(match, g1, g2){return g1+g2.replace(at1, by2).replace(at2, by2).replace(at3, by2)};
        } else {
            var handleAttr = function(match, g1, g2){return g1+g2.replace(at1, by2).replace(at2, by2)};
    }
        html = html.replace(selector, function(match){
             return match.replace(re1, handleAttr).replace(re2, handleAttr);
        });
    }

    /* <meta http-equiv=refresh content="  ; url= " > */
    cri("<meta"+any+att+"http-equiv\s*=\s*(?:\""+ae("refresh")+"\""+any+">|'"+ae("refresh")+"'"+any+">|"+ae("refresh")+"(?:"+ae(" ")+any+">|>))", "content", ae("url")+s+ae("=")+s, "i");

    cr("<"+any+att+"href\s*="+any+">", "href"); /* Linked elements */
    cr("<"+any+att+"src\s*="+any+">", "src"); /* Embedded elements */

    cr("<object"+any+att+"data\s*="+any+">", "data"); /* <object data= > */
    cr("<applet"+any+att+"codebase\s*="+any+">", "codebase"); /* <applet codebase= > */

    /* <param name=movie value= >*/
    cr("<param"+any+att+"name\s*=\s*(?:\""+ae("movie")+"\""+any+">|'"+ae("movie")+"'"+any+">|"+ae("movie")+"(?:"+ae(" ")+any+">|>))", "value");

    cr(/<style[^>]*>(?:[^"']*(?:"[^"]*"|'[^']*'))*?[^'"]*(?:<\/style|$)/gi, "url", "\s*\(\s*", "", "\s*\)"); /* <style> */
    cri("<"+any+att+"style\s*="+any+">", "style", ae("url")+s+ae("(")+s, 0, s+ae(")"), ae(")")); /*< style=" url(...) " > */
    return html;
}

A short summary of the private functions:

私有函数的简短摘要：

• rel_to_abs(url)- Converts relative / unknown URLs to absolute URLs
• replace_all_rel_by_abs(html)- Replaces all relevant occurences of URLs within a string of HTML by absolute URLs.
  1. ae- Any Entity - Returns a RE-pattern to deal with HTML entities.
  2. by- replace by- This short function request the actual url replace (rel_to_abs). This function may be called hundreds, if not thousand times. Be careful to not add a slow algorithm to this function (customisation).
  3. cr- Create Replace - Creates and executes a search-and-replace.
     Example: href="..."(within any HTML tag).
  4. cri- Create Replace Inline - Creates and executes a search-and-replace.
     Example: url(..)within the all styleattribute within HTML tags.

• rel_to_abs(url)- 将相对/未知 URL 转换为绝对 URL
• replace_all_rel_by_abs(html)- 用绝对 URL 替换 HTML 字符串中所有相关的 URL。
  1. ae-一个纽约èntity -返回RE模式来处理HTML实体。
  2. by-取代的-这短短的功能要求的实际的URL替换（rel_to_abs）。这个函数可能会被调用数百次，如果不是数千次的话。注意不要给这个函数添加一个缓慢的算法（定制）。
  3. cr- çreate řE放置-创建并执行一个搜索和替换。
     示例：（href="..."在任何 HTML 标签内）。
  4. cri- çreate řE放置予n第-创建并执行一个搜索和替换。
     示例：url(..)在styleHTML 标签内的 all属性内。

Test case

测试用例

Open any page, and paste the following bookmarklet in the location bar:

打开任何页面，然后将以下书签粘贴到地址栏中：

javascript:void(function(){var s=document.createElement("script");s.src="http://rob.lekensteyn.nl/rel_to_abs.js";document.body.appendChild(s)})();

The injected code contains the two functions, as defined above, plus the test case, shown below. Note: The test case does notmodify the HTML of the page, but shows the parsed results in a textarea (optionally).

注入的代码包含上面定义的两个函数，以及测试用例，如下所示。注意：测试用例不会修改页面的 HTML，而是在 textarea（可选）中显示解析结果。

var t=(new Date).getTime();
  var result = replace_all_rel_by_abs(document.documentElement.innerHTML);
  if(confirm((new Date).getTime()-t+" milliseconds to execute\n\nPut results in new textarea?")){
    var txt = document.createElement("textarea");
    txt.style.cssText = "position:fixed;top:0;left:0;width:100%;height:99%"
    txt.ondblclick = function(){this.parentNode.removeChild(this)}
    txt.value = result;
    document.body.appendChild(txt);
}

See also:

也可以看看：

• Answer: Parsing and sanitising HTML strings

• 答案：解析和清理 HTML 字符串

回答by tuomassalo

A reliable way to convert urls from relative to absolute is to use the built-in urlmodule.

将 url 从相对转换为绝对的一种可靠方法是使用内置url模块。

Example:

例子：

var url = require('url');
url.resolve("http://www.example.org/foo/bar/", "../baz/qux.html");

>> gives 'http://www.example.org/foo/baz/qux.html' 

回答by liberborn

This is Rob W answer "Advanced HTML string replacement functions"in current thread plus some code re-factoring from me to make JSLint happy.

这是Rob W在当前线程中回答“高级 HTML 字符串替换函数”加上我重构的一些代码，以使 JSLint 满意。

I should post it as answer's comment but I don't have enough reputation points.

我应该将其发布为答案的评论，但我没有足够的声望点。

/*jslint browser: true */
/*jslint regexp: true */
/*jslint unparam: true*/
/*jshint strict: false */

/**
 * convertRelToAbsUrl
 *
 * https://stackoverflow.com/a/7544757/1983903
 * 
 * @param  {String} url
 * @return {String} updated url
 */
function convertRelToAbsUrl(url) {
    var baseUrl = null;

    if (/^(https?|file|ftps?|mailto|javascript|data:image\/[^;]{2,9};):/i.test(url)) {
        return url; // url is already absolute
    }

    baseUrl = location.href.match(/^(.+)\/?(?:#.+)?$/)[0] + '/';

    if (url.substring(0, 2) === '//') {
        return location.protocol + url;
    }
    if (url.charAt(0) === '/') {
        return location.protocol + '//' + location.host + url;
    }
    if (url.substring(0, 2) === './') {
        url = '.' + url;
    } else if (/^\s*$/.test(url)) {
        return ''; // empty = return nothing
    }

    url = baseUrl + '../' + url;

    while (/\/\.\.\//.test(url)) {
        url = url.replace(/[^\/]+\/+\.\.\//g, '');
    }

    url = url.replace(/\.$/, '').replace(/\/\./g, '').replace(/"/g, '%22')
            .replace(/'/g, '%27').replace(/</g, '%3C').replace(/>/g, '%3E');

    return url;
}

/**
 * convertAllRelativeToAbsoluteUrls
 *
 * https://stackoverflow.com/a/7544757/1983903
 * 
 * @param  {String} html
 * @return {String} updated html
 */
function convertAllRelativeToAbsoluteUrls(html) {
    var me = this,
        att = '[^-a-z0-9:._]',
        entityEnd = '(?:;|(?!\d))',
        ents = {
            ' ' : '(?:\s|&nbsp;?|&#0*32' + entityEnd + '|&#x0*20' + entityEnd + ')',
            '(' : '(?:\(|&#0*40' + entityEnd + '|&#x0*28' + entityEnd + ')',
            ')' : '(?:\)|&#0*41' + entityEnd + '|&#x0*29' + entityEnd + ')',
            '.' : '(?:\.|&#0*46' + entityEnd + '|&#x0*2e' + entityEnd + ')'
        },
        charMap = {},
        s = ents[' '] + '*', // short-hand for common use
        any = '(?:[^>\"\']*(?:\"[^\"]*\"|\'[^\']*\'))*?[^>]*',
        slashRE = null,
        dotRE = null;

    function ae(string) {
        var allCharsLowerCase = string.toLowerCase(),
            allCharsUpperCase = string.toUpperCase(),
            reRes = '',
            charLowerCase = null,
            charUpperCase = null,
            reSub = null,
            i = null;

        if (ents[string]) {
            return ents[string];
        }

        for (i = 0; i < string.length; i++) {
            charLowerCase = allCharsLowerCase.charAt(i);
            if (charMap[charLowerCase]) {
                reRes += charMap[charLowerCase];
                continue;
            }
            charUpperCase = allCharsUpperCase.charAt(i);
            reSub = [charLowerCase];
            reSub.push('&#0*' + charLowerCase.charCodeAt(0) + entityEnd);
            reSub.push('&#x0*' + charLowerCase.charCodeAt(0).toString(16) + entityEnd);

            if (charLowerCase !== charUpperCase) {
                reSub.push('&#0*' + charUpperCase.charCodeAt(0) + entityEnd);
                reSub.push('&#x0*' + charUpperCase.charCodeAt(0).toString(16) + entityEnd);
            }
            reSub = '(?:' + reSub.join('|') + ')';
            reRes += (charMap[charLowerCase] = reSub);
        }
        return (ents[string] = reRes);
    }

    function by(match, group1, group2, group3) {
        return group1 + me.convertRelToAbsUrl(group2) + group3;
    }

    slashRE = new RegExp(ae('/'), 'g');
    dotRE = new RegExp(ae('.'), 'g');

    function by2(match, group1, group2, group3) {
        group2 = group2.replace(slashRE, '/').replace(dotRE, '.');
        return group1 + me.convertRelToAbsUrl(group2) + group3;
    }

    function cr(selector, attribute, marker, delimiter, end) {
        var re1 = null,
            re2 = null,
            re3 = null;

        if (typeof selector === 'string') {
            selector = new RegExp(selector, 'gi');
        }

        attribute = att + attribute;
        marker = typeof marker === 'string' ? marker : '\s*=\s*';
        delimiter = typeof delimiter === 'string' ? delimiter : '';
        end = typeof end === 'string' ? '?)(' + end : ')(';

        re1 = new RegExp('(' + attribute + marker + '")([^"' + delimiter + ']+' + end + ')', 'gi');
        re2 = new RegExp('(' + attribute + marker + '\')([^\'' + delimiter + ']+' + end + ')', 'gi');
        re3 = new RegExp('(' + attribute + marker + ')([^"\'][^\s>' + delimiter + ']*' + end + ')', 'gi');

        html = html.replace(selector, function (match) {
            return match.replace(re1, by).replace(re2, by).replace(re3, by);
        });
    }

    function cri(selector, attribute, front, flags, delimiter, end) {
        var re1 = null,
            re2 = null,
            at1 = null,
            at2 = null,
            at3 = null,
            handleAttr = null;

        if (typeof selector === 'string') {
            selector = new RegExp(selector, 'gi');
        }

        attribute = att + attribute;
        flags = typeof flags === 'string' ? flags : 'gi';
        re1 = new RegExp('(' + attribute + '\s*=\s*")([^"]*)', 'gi');
        re2 = new RegExp("(" + attribute + "\s*=\s*')([^']+)", 'gi');
        at1 = new RegExp('(' + front + ')([^"]+)(")', flags);
        at2 = new RegExp("(" + front + ")([^']+)(')", flags);

        if (typeof delimiter === 'string') {
            end = typeof end === 'string' ? end : '';
            at3 = new RegExp('(' + front + ')([^\"\'][^' + delimiter + ']*' + (end ? '?)(' + end + ')' : ')()'), flags);
            handleAttr = function (match, g1, g2) {
                return g1 + g2.replace(at1, by2).replace(at2, by2).replace(at3, by2);
            };
        } else {
            handleAttr = function (match, g1, g2) {
                return g1 + g2.replace(at1, by2).replace(at2, by2);
            };
        }
        html = html.replace(selector, function (match) {
            return match.replace(re1, handleAttr).replace(re2, handleAttr);
        });
    }

    cri('<meta' + any + att + 'http-equiv\s*=\s*(?:\"' + ae('refresh')
        + '\"' + any + '>|\'' + ae('refresh') + '\'' + any + '>|' + ae('refresh')
        + '(?:' + ae(' ') + any + '>|>))', 'content', ae('url') + s + ae('=') + s, 'i');

    cr('<' + any + att + 'href\s*=' + any + '>', 'href'); /* Linked elements */
    cr('<' + any + att + 'src\s*=' + any + '>', 'src'); /* Embedded elements */

    cr('<object' + any + att + 'data\s*=' + any + '>', 'data'); /* <object data= > */
    cr('<applet' + any + att + 'codebase\s*=' + any + '>', 'codebase'); /* <applet codebase= > */

    /* <param name=movie value= >*/
    cr('<param' + any + att + 'name\s*=\s*(?:\"' + ae('movie') + '\"' + any + '>|\''
        + ae('movie') + '\'' + any + '>|' + ae('movie') + '(?:' + ae(' ') + any + '>|>))', 'value');

    cr(/<style[^>]*>(?:[^"']*(?:"[^"]*"|'[^']*'))*?[^'"]*(?:<\/style|$)/gi,
        'url', '\s*\(\s*', '', '\s*\)'); /* <style> */
    cri('<' + any + att + 'style\s*=' + any + '>', 'style',
        ae('url') + s + ae('(') + s, 0, s + ae(')'), ae(')')); /*< style=" url(...) " > */

    return html;
}

回答by Geordie

From a comment by Rob W above about the base tag I wrote an injection function:

从 Rob W 上面关于基本标签的评论中，我写了一个注入函数：

function injectBase(html, base) {
  // Remove any <base> elements inside <head>     
  html = html.replace(/(<[^>/]*head[^>]*>)[\s\S]*?(<[^>/]*base[^>]*>)[\s\S]*?(<[^>]*head[^>]*>)/img, " ");

  // Add <base> just before </head>  
  html = html.replace(/(<[^>/]*head[^>]*>[\s\S]*?)(<[^>]*head[^>]*>)/img, " " + base + " ");  
  return(html);
}

回答by Jani Hartikainen

If you use a regex to find all non-absolute URLs, you can then just prefix them with the current URL and that should be it.

如果您使用正则表达式来查找所有非绝对 URL，那么您只需在它们前面加上当前 URL 的前缀就可以了。

The URLs you need to fix would be ones which don't start either with a /or http(s)://(or other protocol markers, if you care about them)

您需要修复的 URL 不以 a/或http(s)://（或其他协议标记，如果您关心它们）开头

As an example, let's say you're scraping http://www.example.com/. If you encounter a relative URL, let's say foo/bar, you would simply prefix the URL being scraped to it like so: http://www.example.com/foo/bar

例如，假设您正在抓取http://www.example.com/. 如果您遇到相对 URL，比方说foo/bar，您只需将要抓取的 URL 作为前缀，如下所示：http://www.example.com/foo/bar

For a regex to scrape the URLs from the page, there are probably plenty of good ones available if you google a bit so I'm not going to start inventing a poor one here :)

对于从页面中抓取 URL 的正则表达式，如果您稍微谷歌一下，可能有很多好的可用，所以我不会在这里开始发明一个糟糕的 :)