| CARVIEW |
Navigation Menu
GitHub Community
-
|
There is useful content in Github wikis. It's unfortunate that external search engines are banned People have put time into building their wikis and it's sad that the information is almost impossible to discover - unless you think to try Github search. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 27 -
😄 2 -
👀 3
Replies: 16 comments · 53 replies
-
|
For reference here's an old issue from the old isaacs repo: isaacs/github#1683 |
Beta Was this translation helpful? Give feedback.
All reactions
-
❤️ 3
-
|
Another old discussion, which pretty much is a repeat: https://github.community/t/request-github-open-up-project-wiki-pages-to-web-indexes/122096 |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
FYI @EdVassie |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
It also blocks archive.org. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
they are not banned, robots.txt does nothing to their ability to crawl. It is just that they should not crawl and they don't. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👎 6 -
😕 2
-
|
I'm not sure what you mean, but line 11 in https://github.com/robots.txt is literally this: |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Disallow: //wiki is just a directive... if you build your crawler to ignore directives... there is no tech involved trying to stop your crawl provided by the robots.txt file. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👎 7 -
😕 10
-
|
@timothywcrane the question is not about us writing some crawler to crawl GitHub Wikis (because you could just ignore |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 24
-
|
LIne 11 of https://github.com/robots.txt does not mention wiki anymore Actually wiki is no longer mentioned at https://github.com/robots.txt? |
Beta Was this translation helpful? Give feedback.
All reactions
-
🎉 2 -
👀 1
-
|
Sent in a PR to remove the statement in the docs: github/docs#10836 . |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Great! Thank you GH and @nelsonjchen for the service you provided. We now moved to GH pages, but for sure this is a step in the right direction |
Beta Was this translation helpful? Give feedback.
All reactions
-
❤️ 1
-
|
No kidding! It's nice to see they opted to change the policy after nearly a decade. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
|
Oh, um, nevermind. 😖 |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
The nightmare continues. |
Beta Was this translation helpful? Give feedback.
All reactions
-
😄 7
-
|
Hi everyone 👋 we have intentionally excluded |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 7 -
👎 74 -
😕 9 -
🚀 1 -
👀 3
-
|
I can see a counter argument for the star count criteria. It seems to me that the projects that most need to be search-enabled are those that aren't already well known. A project with a high star count probably already has links on lots of web pages. Now it is true, absolutely true, that the high-star projects ought to be indexed. But the place where being search-enabled can provide the most benefit is for those projects that serve a niche application and haven't yet been discovered by a lot of developers. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 12 -
👀 2
-
|
Qweqwe4844 |
Beta Was this translation helpful? Give feedback.
All reactions
-
😕 3
-
|
Okay, I hope in the nearest future we will begin to see some changes... |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Update: ~500 star wikis that are non-publically-editable appear to be indexable now: https://github.com/IgniteUI/igniteui-angular/wiki No
|
Beta Was this translation helpful? Give feedback.
All reactions
-
👀 2
-
Please announce changes before. We are currently adding a statistics wiki to our git profile which is also found on our website. If this would become crawlable for search engines, it would cause some major SEO issues (duplicate content) for us. However, allowing users to add html tags like rel="canonical" to wiki pages would solves this issue. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Wikis are not just the problem: google simply ignores ANY page in github, even readme.md! I can never find any suitable result in google coming from github. The only way for searching in github is using github search engine. But non-developers do not know github, so why should they search there? |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Maybe you could just add |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
it's just an example; github never comes up in generic google searches, although it contains thousands of programs, not just sources. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
perhaps things changed meanwhile, but https://www.google.com/search?q=%22lunar+lander+game%22+%22github%22+%22wiki%22 |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
@dimpase https://github.com/orgs/community/discussions/4992#discussioncomment-3563126 |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Fortunately there are some search engines that ignore the robot exclusion rule. They will in turn allow DuckDuckGo, Bing, and other bots to crawl their cached version of GitHub wiki pages - allowing us to find stuff on GitHub. Its nice that someone cares more about finding content on the Internet Now that the disallowing of robots has been rendered moot: lets do the right thing and remove any |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 2 -
❤️ 3
-
|
Not sure what the conclusion of this discussion is, but we're facing this problem: Around a year ago, I renamed my org from github.com/Otykier to github.com/TabularEditor. Now it seems, that none of the repo issues pages are being crawled. For example, I can take a literal quote from this issue (which was created more than a year ago) and get no hits on Google. This goes for all issues - none of them seem to be crawled, making it very hard for users of our software to find solutions to problems they encounter. We can still get to the main page by searching for "github tabular editor 3", for example, but no search results from the issues pages. Is there some org or repo setting we're missing? I am 90% sure that it used to work before we renamed the org - could that have something to do with it? Thanks! |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
I am thinking to move away from GitHub with this robots.txt policy :-( |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
alternatives? |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
I switched over to SourceForge. |
Beta Was this translation helpful? Give feedback.
All reactions
-
😄 3
-
|
SF is not an alternative - they do not offer anything like GitHub actions, their GUI to work with code online is not existing, AFAIK. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
We moved away from wikis to a website (hosted by GitHub Pages). I'd also consider GitLab an alternative. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 3
-
|
I am sure GitHub can easily arrange with at least one of the big search
engine to be well-crawled and indexed.
And this will give said engine a considerable boost in popularity.
…On Wed, 1 Mar 2023, 09:04 jumpjack, ***@***.***> wrote:
alternatives?
—
Reply to this email directly, view it on GitHub
<#4992 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJXYHBJHJMFBPVOBZ4XUQTWZ37HBANCNFSM5B3UN4OQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
All reactions
-
|
GitHub is now Microsoft owned. Don't know if they want their Azure in competition to GitHub. That is the reason that I move over to an alternative. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
azure does not do what github does, there is no competition; github uses azure to host everything they need, bringing extra business to azure. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 2
-
|
We are seeing github wiki pages now being indexed - has this changed permanently or is it temporarily? |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
|
I asked OpenAI: https://chat.openai.com/c/308cf4b9-2759-41ca-a243-740279bbe1a7 |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
You have to press "Share Conversation". Copy/pasting the URL won't work. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
I'm getting fairly desperate about various GitHub behaviors, but this is cherry on cake :( My repo has 54K stars and in spite of a comment above suggesting that 500+ stars repos wikis would be indexed it doesn't seem like it is. EDIT I WOULD NEED TO MAKE MY WIKI UNEDITABLE BY USERS TO DO SO ?!!! |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
It must also not permit non-collaborators to edit in addition to the star requirement. I've noticed this criteria while developing that mirror thing. You can find a bit more info on the front page of that mirroring service https://github-wiki-see.page as discussions are always a bit fragmented. Does your repo also meet that criteria? |
Beta Was this translation helpful? Give feedback.
All reactions
-
❤️ 1
-
|
It doesn’t, sorry i amended my post afterwards. It seems the betray the desirable purpose of my wiki to restrict edits. Thank you for your work on this Nelson . |
Beta Was this translation helpful? Give feedback.
All reactions
-
❤️ 1
-
|
I need to clarify this didn't really help as well as I'd hoped, in spite of adding links here and there, I don't often stumble on the right mirrored info from a search engine. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
I have a wiki in my repository containing documentation and coding standards for my repository. |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
(reposting some details not as a threaded reply) @dipree: This is really harmful and disappointing, actively discouraging people investing in Wiki. I can't comprehend why it hasn't been solved. I understand that abuses are a problem, but a single "Allow only to collaborators" options is very limiting, other mitigating options should be possible (see below). I have 56K+ stars, a growing Wiki https://github.com/ocornut/imgui/wiki and I would like other people than me to work on it, but right now its contents is not indexed in spite of best effort to even link to https://github-wiki-see.page etc. meaning people are having difficulty finding the information they need, and keep asing us the same things, lowering the quality of their software experience and hogging support/dev resources. :( Can't github implement some alternative options? Suggestions in order of simplicity:
If even the first option was added, I would cave in and lock the wiki tomorrow and add likely contributors + infos to request. I suspect that part of your underlying thought might be that by requiring developers to grant full access to a repository to allow people edit a wiki, it is ensuring we don't add too many people? Can't exceptions be made to well behaving projects? |
Beta Was this translation helpful? Give feedback.
All reactions
-
|
Unfortunately, the system, for better or worse, isn't really ranking that high on search engines nowadays. Even I find it pretty darn hard to use Google to look up openpilot wiki information which was the original impetus. I'm thinking of maybe producing some sort of self mirroring toolkit for projects. Like a recipe for what the shellcheck project did. The existing system will still be there for general wikis though. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 3
-
|
Didn't realize that wiki's are not indexed. I just stumbled upon my own wiki in a search result hosted via GitHub Wiki Search Engine Enablement (GHWSEE). (thanks for this useful resource!) I had started putting things into wiki specifically so it could be indexed for others, but it appears not to be. That's unfortunate. |
Beta Was this translation helpful? Give feedback.
Hi everyone 👋 we have intentionally excluded
Disallow: /*/wiki*from therobots.txt. However, we have also introduced anx-robots-tag: nonein the http response header of Wiki pages. As a result, Wikis are still not visible to search engine crawlers. Why have we done this? Abusive behavior in Wikis had a negative impact on our search engine ranking and therefore we had to exclude Wikis from getting crawled to mitigate the effects. We are now investigating options how we can open the gates again so that everyone can benefit from the great information documented in Wikis. At this point, we have not decided on whether or when we will allow Wikis to be crawled again, but we are actively revie…