--- metadata: - name: generator content: Diplodoc Platform v5.64.0 alternate: - https://mobi.yandex.com/support/webmaster/en/robot-workings/allow-disallow.md - https://mobi.yandex.com/support/webmaster/tr/robot-workings/allow-disallow.md - href: https://mobi.yandex.com/support/webmaster/en/robot-workings/allow-disallow.md type: text/markdown title: Markdown version - href: https://mobi.yandex.com/support/webmaster/en/llms.txt?revision=r21762422 rel: describedby --- > **Documentation Index:** Fetch the complete configuration index at https://mobi.yandex.com/support/webmaster/en/llms.txt?revision=r21762422 # Disallow and Allow directives ## Disallow {#disallow} Use this directive to prohibit crawling of sections or individual pages of a site. For example: - Pages that contain confidential data. - Pages with site search results. - Site traffic statistics. - Duplicate pages. - Various logs. - Database service pages. {% note info %} When selecting a directive for pages that need to be left out from the search, if their addresses contain GET parameters, it's recommended to use the [Clean-param](https://mobi.yandex.com/support/webmaster/en/robot-workings/clean-param.md) directive rather than Disallow. If you use Disallow, you may not be able to identify duplicate link URLs without the parameter and send some metrics of forbidden pages. {% endnote %} Examples: ```html User-agent: Yandex Disallow: / # prohibits crawling of the entire site User-agent: Yandex Disallow: /catalogue # prohibits crawling of pages whose addresses start with /catalogue User-agent: Yandex Disallow: /page? # prohibits crawling of pages whose URLs contain parameters ``` ## Allow {#allow} This directive allows crawling of sections or individual pages of a site. Examples: ```no-highlight User-agent: Yandex Allow: /cgi-bin Disallow: / # prohibits downloading of everything except pages # starting with “/cgi-bin” ``` ```no-highlight User-agent: Yandex Allow: /file.xml # allows downloading of the file.xml file ``` {% note info %} Empty line breaks aren't allowed between the `User-agent`, `Disallow` and `Allow` directives. {% endnote %} ## Combining directives {#simultaneous} The `Allow` and `Disallow` directives from the corresponding `User-agent` block are sorted according to URL prefix length (from shortest to longest) and applied in order. If several directives match a particular site page, the robot selects the last one in the sorted list. This way the order of directives in the `robots.txt` file doesn't affect the way they are used by the robot. {% note info %} If there is a conflict between two directives with prefixes of the same length, the `Allow` directive takes precedence. {% endnote %} ```no-highlight # Source robots.txt: User-agent: Yandex Allow: / Allow: /catalog/auto Disallow: /catalog # Sorted robots.txt: User-agent: Yandex Allow: / Disallow: /catalog Allow: /catalog/auto # prohibits downloading of pages starting with “/catalog”, # but allows downloading of pages starting with “/catalog/auto”. ``` Common example: ```no-highlight User-agent: Yandex Allow: /archive Disallow: / # allows everything containing “/archive”, the rest is prohibited User-agent: Yandex Allow: /obsolete/private/*.html$ # allows html files # in “/obsolete/private/...” Disallow: /*.php$ # prohibits all “*.php” on this site Disallow: /*/private/ # prohibits all subpaths containing # “/private/”, but Allow above negates # part of this prohibition Disallow: /*/old/*.zip$ # prohibits all “*.zip” files containing # in “/old/” User-agent: Yandex Disallow: /add.php?*user= # prohibits all “add.php?” scripts with the “user” parameter ``` ## Allow and Disallow directives without parameters {#parameterless} If directives don't contain parameters, the robot handles the data as follows: ```no-highlight User-agent: Yandex Disallow: # same as Allow: / User-agent: Yandex Allow: # isn’t taken into account by the bot ``` ## Using the special characters * and $ {#asterisk-dollar} You can use special characters when specifying the paths of the Allow and Disallow directives `*` and `$` to set certain regular expressions. The `*` character indicates any sequence of characters (or none). Examples: ```no-highlight User-agent: Yandex Disallow: /cgi-bin/*.aspx # prohibits “/cgi-bin/example.aspx” # and “/cgi-bin/private/test.aspx” Disallow: /*private # prohibits both “/private” # and “/cgi-bin/private” ``` By default, the `*` character is appended to the end of every rule described in the `robots.txt` file. Example: ```no-highlight User-agent: Yandex Disallow: /cgi-bin* # blocks access to pages # starting with “/cgi-bin” Disallow: /cgi-bin # the same ``` To cancel `*` at the end of the rule, use the `$` character, for example: ```no-highlight User-agent: Yandex Disallow: /example$ # prohibits “/example”, # but allows “/example.html” ``` ```no-highlight User-agent: Yandex Disallow: /example # prohibits both “/example” # and “/example.html” ``` The `$` character doesn't forbid the `*` at the end, that is: ``` User-agent: Yandex Disallow: /example$ # only prohibits “/example” Disallow: /example*$ # same as “Disallow: /example” # prohibits both /example.html and /example ``` ## Processing the # character {#hash} According to the standard, you should insert a blank line before every User-agent directive. The `#` character designates commentary. Everything following this character, up to the first line break, is disregarded. Pages with addresses like `https://example.com/page#part_1` are not indexed by the search bot and will be crawled at the `https://example.com/page` address. Therefore, it is okay to specify the page address without the anchor in the directive. If you do not take this feature into account and write a disallow directive with the `#` character, it may block the entire site from indexing. For example, a directive in the `Disallow: /#` format will be interpreted by the search system as `Disallow: /`, i.e., a complete ban on indexing. ## Examples of how directives are interpreted {#interpre} ```no-highlight User-agent: Yandex Allow: / Disallow: / # everything is allowed User-agent: Yandex Allow: /$ Disallow: / # everything is prohibited except the home page User-agent: Yandex Disallow: /private*html # prohibits “/private*html”, #, “/private/test.html”, “/private/html/test.aspx”, etc. User-agent: Yandex Disallow: /private$ # only prohibits “/private” User-agent: * Disallow: / User-agent: Yandex Allow: / # since the Yandex bot # selects entries that include “User-agent:”, # everything is allowed ```
{% cut "Contact support" %}
{% endcut %}



You can also go to

[![](../_assets/logo/logo-pic.svg)](https://webmaster.yandex.com/) [![](../_assets/logo/logo-en.svg)](https://webmaster.yandex.com/)