Mish's Global Economic Trend Analysis

---

• Obama's Nuclear Deal With Iran: Rare Victory for Peace or Foolish Compromise?
• Bank of Canada Admits Recession: Shades of Voldemort; The Solution: More Bubbles; Operation Twist Canadian Style?
• Emanuel Fiddles While Chicago Burns; Public Schools Over the Edge; 9% Cloud Tax on Data Streaming; Emanuel Eyes Property Tax Hikes

Obama's Nuclear Deal With Iran: Rare Victory for Peace or Foolish Compromise? Posted: 17 Jul 2015 03:59 PM PDT Historic Deal On Tuesday, President Obama, World Powers, and Iran agreed on a Historic Nuclear Pact phasing out economic sanctions in return for curbs on Iran's nuclear program. President Barack Obama said the deal with Tehran ensures that "every pathway to a nuclear weapon" has been cut off. "We have stopped the spread of nuclear weapons in this region," he said in an early-morning statement that — in a rare move — was carried on Iranian television Republicans Blast Deal The Bustle reports Republicans Respond To The Iran Deal With Every Bit As Much Vitriol As You'd Expect — And Then Some. "I have said from the beginning of this process that I would not support a deal with Iran that allows the mullahs to retain the ability to develop nuclear weapons, threaten Israel, and continue their regional expansionism and support for terrorism," said Florida Sen. Marco Rubio in a statement on Tuesday. "Based on what we know thus far, I believe that this deal undermines our national security." "I expect that a significant majority in Congress will share my skepticism of this agreement and vote it down," said Rubio, calling on the next president to effectively wipe the slate clean once again and trash the deal. "It will … be left to the next President to return us to a position of American strength and re-impose sanctions on this despicable regime …" Rubio's GOP counterparts were equally displeased. "Iran's Supreme Leader should know that a future American president will not be bound by this diplomatic retreat," said Wisconsin Governor Scott Walker in a statement on Tuesday morning. "Undoing the damage caused by this deal won't be easy [but] when the United States leads, and has a President who isn't eager to embrace Iran, the world will follow." As president, on my very first day going forward, I would pull back, I would terminate that bad deal with Iran completely on day one. I would then put in place crippling economic sanctions against Iran, and I'd convince our allies to do the same. This is not a country we should be doing business with. "This is the most dangerous, irresponsible step I've ever seen in the history of watching the Mideast," said South Carolina Senator (and 2016 hopeful) Lindsey Graham in a visit to MSNBC's Morning Joe on Tuesday. "You've ensured that the Arabs will go nuclear … put Israel in the worst possible box [and it] will be a death over time sentence to Israel if they don't push back," added Lindsey, directing his commentary to the president. Democrats Lukewarm Not even the Democrats seemed pleased with the announcement. The Hill reports Dems Hold Judgment on Iran Deal. Sen. Charles Schumer (N.Y.) and other Democrats are taking a wait-and-see approach to the Iran nuclear deal announced by President Obama on Tuesday morning. Schumer, who is poised to lead Senate Democrats in the next Congress, said he would go through the agreement with a "fine-tooth comb" and speak to administration officials. Senate Democratic Leader Harry Reid (Nev.) also stopped short of endorsing the deal. Sen. Bob Menendez (N.J.), a former chairman of the Foreign Relations Committee is the only Senate Democrat so far to publicly criticize the agreement. "The bottom line is: The deal doesn't end Iran's nuclear program — it preserves it," he said in a statement. Menendez's influence in the Democratic caucus has waned, however, since he stepped down as the senior Democrat on Foreign Relations after being indicted on corruption charges. Sen. Michael Bennet (Colo.), the most vulnerable Senate Democratic incumbent of the 2016 election cycle, echoed Reid's caution. "We will carefully scrutinize the terms of this agreement. The stakes are high, and the details of this deal matter," he said. The Republicans hate the deal, and the Democrats seems to be holding their noses while semi-supporting it. Is anyone happy? Bernie Sanders is Happy The Hill reports "One of the most enthusiastic responses in the Democratic caucus came from Sen. Bernie Sanders of Vermont, who is running for the Democratic presidential nomination on a liberal platform." Let me point out that Bernie Sanders is an Independent. He merely caucuses with with the Democratic party. Here is Sanders' statement: "I congratulate President Obama, Secretary Kerry and the leaders of other major nations for producing a comprehensive agreement to prevent Iran from obtaining a nuclear weapon. This is a victory for diplomacy over saber-rattling and could keep the United States from being drawn into another never-ending war in the Middle East." Is Anyone Else Happy? Yes, Me. So is David Stockman, libertarians in general, and anyone, regardless of political party, who can think clearly. I have not thought about it this way before, but if no one in Washington is happy, I probably am. I exempt compromises in which both sides got something and parties are unhappy only because the other side got anything at all. War Mongering Fools It's high time both parties party rid themselves of warmongering fools like Marco Rubio, Scott Walker, Lindsey Graham, and Bob Menendez. Rubio and Walker are particularly troubling. In the past I have praised Walker for taking on the public unions in Wisconsin, but I cannot support anyone with warmongering views as extreme as his. They claim among other things "The US cannot trust Iran". If they had an ounce of common sense, they might ask the question: "Why the hell should Iran trust the US?" The irony should be obvious. Rubio and Walker promised to renege on the deal in advance. The US blew Iraq to smithereens under president Bush for no reason at all. The WOMD threat was a known lie. The Republican party's primary excuse was that Hillary supported the war. This only proves war-mongering idiocy has no political party bounds. The US backs Israel no matter what Israel does. The US, with cooperation from the British, overthrew a democratically elected government in Iran and installed a US puppet suitable to US oil interests. The Iranian coup d'état was named "Operation Boot" in the UK, and "TPAJAX" in the US. US idiocy in Iraq spawned ISIS. The result was that Iran and the US became allies in the fight, but we did not want their help. Politics at any cost got in the way. Lesson for Hypocrites US hypocrites have some nerve bitching we cannot trust Iran, when Iran has every right not to trust the US! I commented on this before, on April 6, 2015 in History Lesson for Hypocrites, Warmongers, and Fools. In 2013, under freedom of information lawsuits, the Guardian reported CIA Admits Role in 1953 Iranian Coup. Netanyahu Lesson from 2002 Here's another history lesson: Who said "If You Take Out Saddam, I Guarantee It Will Have Enormous Positive Reverberations". Here's the answer: Benjamin Netanyahu in 2002. Gingrich vs. Reagan 1985 Anyone recall what Republican House Speaker Newt Gingrich said about Ronald Reagan's meeting with Gorbachev in 1985? The answer is amusing: Gingrich called it "the most dangerous summit for the West since Adolf Hitler met with Neville Chamberlain in 1938 in Munich." War in Iraq And the War with Iraq? In 2003 Defense Secretary Donald Rumsfeld and his deputy Paul Wolfowitz said "the war could be done on the cheap and that it would largely pay for itself." Here we are trillions of dollars later and counting. And ISIS is a direct result of that stupid war with Iraq. Fighting ISIS In 2013 the Washington Post stated Iraq and Afghanistan Wars Costs Top $4 Trillion. The Washington Post forgot to factor in ISIS. Did Iran Threaten to "Wipe Israel off the Map"? The Hitler Card players in both parties keep referring to an alleged statement by Iran's Mahmoud Ahmadinejad to "Wipe Israel Off the Map". The problem is twofold. Ahmadinejad was not the power broker in Iran. The translation was not even correct. Ahmadinejad was actually referring to a statement made by leader Khamenei in which Khamenei said in Persian "Een rezhim-i eshghalgar-i Quds bayad az sahneh-i ruzgar mahv shaved." This means, "This occupation regime over Jerusalem must vanish from the arena of time." Anonymous wire service translators rendered Khomeini as saying that Israel "must be wiped off the face of the map." The myth still persists to this day. President Bush, and Even President Obama have used it when it suits their purpose. And of course, every time that I have spoken favorably of the agreement, I hear the same "wipe the map" nonsense from readers. Rare Victory for Peace Due to political divisiveness, political leaders are never willing to say the other side did anything right. That's why I will never be a political leader. I bash Obama often enough, and for many things. But I equally bash the warmongers who have been wrong every which way, and virtually every time! This was a rare victory for peace. I am proud of the fact that I stood against the Vietnam War, I was against the War in Iraq, and I have been on the right side of the Iran debate from the beginning. I echo Ronald Reagan's former budget director David Stockman who says All Praise To Barack Obama For Stiffing The War Party—- Peace Is Finally Being Given A Chance. I even go one step further and encourage President Obama to resume normal relations with Iran. Only good can come from such talks. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

Bank of Canada Admits Recession: Shades of Voldemort; The Solution: More Bubbles; Operation Twist Canadian Style? Posted: 17 Jul 2015 11:16 AM PDT The Bank of Canada admitted on Wednesday that Canada was in Recession. Well sort of. Bank of Canada Governor Stephen Poloz is afraid to speak the "R-Word". Instead, Poloz phrased it this way: "Real GDP is now projected to have contracted modestly in the first half of the year." Shades of Voldemort Poloz further went out of his way to state the "R-word is unhelpful". Pressed by reporters to just come out and use the R-word, Poloz dug in. "I just find the discussion quite unhelpful," he sniffed. "It's especially unhelpful when what has happened to the economy is very narrowly defined." Recession an Easy Call Calling the Canadian recession was one of the easiest calls ever. I did so on January 31, in Canada in Recession, US Will Follow in 2015. What made the Canadian recession easy to spot was the Canadian yield curve inverted out to three years following a surprise rate cut by the Bank of Canada on January 21. It remains to be seen if the US follows. The US contracted in the first quarter, but the second quarter rebound was a bit stronger than I expected. On January 21, in response to the surprise cut, I wrote Canadian Recession Coming Up: Yield Curve Inverts Following Unexpected Rate Cut; Loonie at Six-Year Low. In that post I awarded Canada the "Blue Ribbon" for the first yield curve inversion of any major country following the great financial crisis. I am not aware of anyone else noticing or commenting on the yield-curve inversion at the time. Denial Sets In Already Two consecutive quarters of GDP contraction are a sufficient but not necessary condition for recession. Indeed the NBER, the official arbiter of US recessions, has called the start of recessions in quarters in which GDP was positive. I have not seen anyone deny a recession when there has been 2 quarters of negative growth, "mild" or not, until now. The Globe and Mail amusingly claims Canada's Fall into Recession Far from a Sure Thing. The question of whether Canada slipped into a recession in the first half of 2015 may depend on our definition of "recession." And most of us have been using the wrong one, says one of the country's top experts on measuring business cycles. Many commentators pointed at the Bank of Canada's revelation this week that the Canadian economy likely contracted in the second quarter as evidence that the country met the "technical" definition of a recession: two consecutive quarters of declining gross domestic product. Indeed, much was made of Bank of Canada Governor Stephen Poloz's unwillingness to even utter the word "recession" after the central bank's interest-rate cut Wednesday. The Solution: More Bubbles Apparently, if you don't like the word recession, you not only don't say it, you change the meaning of it. Then you need to do something to make the problem go away. The tried and true central bank method the world over is to blow more bubbles. Pater Tenebrarum at the Acting Man blog offered his well thought out take in Bank of Canada Decides More Bubble-Blowing is Needed You Can't Keep the Printing Press Idle for too Long … We have recently portrayed Canada's new central bank governor Stephen Poloz, to whom we have alternately referred to as a comedian and a delusional bubble blower. This may perhaps strike some readers as uncharitable; then again, central economic planning bureaucrats should be fair game, especially as nearly all of them are slaves to hoary inflationism and are apodictically certain to do grave damage to the economy, based on economic theories that at best deserve to be called a form of voodoo. It's really that bad. As readers may recall, Mr. Poloz has continued where his fellow bubble-blower and predecessor Mark Carney left off, by keeping the bubble blown with all his might. We imagine he may be a bit intimidated by the truly daunting size the combined real estate and consumer credit bubbles have attained in Canada. To call them monuments to monetary megalomania would be an understatement. Among developed nations, only the bubbles in a few Scandinavian countries and Australia can hold a candle to them. We were therefore decidedly unsurprised when it emerged yesterday that the Bank of Canada has cut rates again – apparently the Canadian economy has entered an official recession, which must however not be mentioned. Canadian Household Debt Bubble for the Record Books The above chart reads "This is a credit bubble for the history books. The 2008 Bubble couldn't stop it from growing, so when it finally does implode, it will probably be quite spectacular. This is likely to go hand-in-hand with the demise of the equally egregious housing bubble." Chart courtesy of Pater Tenebrarum. Pater also offered these pertinent comments. Several years go already about one third of Canadians reported in a survey that their debt worries caused them to have sleeping problems. We can certainly believe it. Stephen Poloz, comedian, gifted pantomime, delusional bubble blower and hoary inflationist, sprung directly from the John Law School of economics (just as the rest of the developed world's central planners), is trying to prolong the bubble's life some more. So far, he has been successful, but he is riding a tiger. The sooner such policies are discredited, the better it will be for all of us, even though the process is likely to be painful for many. Change 'likely' to 'will be' and many to 'nearly everyone, but especially those who got into the real estate bubble in the latter stages' and Pater is precisely correct. Yield Curve Still Inverted By the way, portions of the Canadian Yield Curve are still inverted, albeit very slightly. I compiled this list of rates today from Investing.Com. 30-Year: 2.242% 10-Year: 1.562% 7-Year: 1.154% 5-Year: 0.705% 4-Year: 0.446% 3-Year: 0.383% 2-Year: 0.423% 1-Year: 0.445% 6-Month: 0.435% 3-Month: 0.400% 1-Month: 0.390% Note the inversion: 1M, 3M, 6M, 1Y, and 2Y yields are all higher than the 3Y yield. Operation Twist, Canada Style I smell an "Operation Twist" type move by the Canadian central bank to rectify this horrific "recession-signaling" condition. If so, the sweet spot for banks and hedge funds to front-run the trade appears to be 5Y or 7Y notes. Then again, some banks may already be in on it. I suspect some of them were not at all surprised by the surprise announcement in January. That's simply the way the system "works". Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

Emanuel Fiddles While Chicago Burns; Public Schools Over the Edge; 9% Cloud Tax on Data Streaming; Emanuel Eyes Property Tax Hikes Posted: 16 Jul 2015 11:13 PM PDT Chicago public schools are effectively bankrupt. Its pension plans are countless billions of dollars in the hole, the worst in the nation. And Chicago bonds carry a junk rating, the worst of any major US city. There is plenty to fix. But instead of addressing the enormous problems Chicago faces, Mayor Rahm Emanuel opts to nickel and dime residents and businesses to death, scaring away companies away in the process. He also threatens property tax hikes that rate to be enormous. There's much more in this article. Make sure you get to the details of how Chicago "funded" its pension plan this year.  Starting from the top ... "9% Cloud Tax on Streaming" In April, Emanuel announced a commitment from technology companies to create 1,000 new jobs. This month, Emanuel rewarded those technology companies with a 9% Cloud Tax on Online Movies, Music, Game, Real Estate Data Feeds. Streaming data? Movies? Real estate Feeds? If you have a Chicago business address expect to pay 9% more to do so thanks to an amusement tax and property lease tax extension. Chicagoans who pay to stream movies and music from services like Netflix and Spotify will now need to fork over an additional 9 percent for the privilege, as will Chicago businesses that pay to use everything from real estate to court databases online, under a decision the city quietly made recently to expand its taxing power. According to the Finance Department changes, the 9 percent amusement tax, which has mostly been tacked onto tickets to concerts and sporting events, also now applies to paid subscriptions for streamed digital music and to streamed rental movies or TV shows, and "for the privilege of participating in games, on-line or otherwise," if the person paying to receive the data is in Chicago. The personal property lease transaction tax expansion also applies to professional services, like electronic property databases real estate agents use, court case databases lawyers rely on and various financial information networks. "I think they have taken the existing lease transaction tax and existing amusement tax as far as you can take them," said Michael Wynne, a partner and attorney in the Chicago office of the law firm Reed Smith. "Let's say I sign up for streaming business data in the city but I have offices throughout the country," Wynne said. "I will definitely make sure my billing goes through a different office. Anything that requires a user to do a search or make a request is subject to the tax. "There is nothing you can access that's not taxable," he said. Minimum Wages Chicago hopes to collect $12 million from the streaming tax. Let's assume it does. How many businesses will take a look at Chicago nickel and diming everyone to death and decide there are better places to do business?  Recall that in December, the Chicago raised its minimum wage to $13.00 by 2019. It's $10 an hour as of July 1, 2015. State Problems Just outside Chicago, there's a place called Illinois. And Illinois is not a good place to do business. Illinois has .... The second-highest property taxes nationwide. The fourth-highest workers' compensation costs in the country. The ninth-highest regulatory burden in the U.S. The ninth-highest overall tax burden nationwide. Forced unionism. Progressives have been angling for a massive income tax hike. The "glory" does not stop there. Please note that Illinois has the most units of local government of any state in the country. According to the U.S. Census Bureau, with 6,963 local governments, Illinois beats its nearest competitor by more than 1,800. Texas is No. 2 with 5,147 local governments. For more details please see Congratulations to Illinois: Most Government Bodies, Most Convicted Governors, Lowest Credit Rating. Chicago Public Schools Over the Edge Anyone living in Chicago has to deal with Chicago schools. The district is both politically and fiscally bankrupt. On June 25, The public school system contract talks ground to a halt. Even though its pension plan is horribly unfunded, someone hatched a plan to borrow $500 million from it, to avoid teacher cuts. Emanuel Eyes Property Taxes The Pension Borrowing Scheme was Shelved and instead, Emanuel has his eyes on your wallet. The Chicago Public Schools and the Chicago Teachers' Pension Fund have ended discussions on a school district proposal to borrow $500 million from the pension to avoid additional layoffs and classroom cuts. The district recently paid a $634 million pension bill that officials said it couldn't afford. As a result, the nation's third-largest school district plans 1,400 layoffs, scaled-back maintenance and reduced transportation. Mayor Rahm Emanuel blames state legislators for worsening the situation and says a property tax increase is up for consideration. Budget Cuts? Where? Instead of facing the fact that the system is bankrupt, Emanuel wants still higher property taxes. The Chicago Public School Budget brags about cutting $55 million. Here is a bit of perspective on those cuts from The Stump. $55 million out of a $5.8 billion budget is not much more than a rounding error. Illinois Budget $4 Billion in the Hole The Chicago school budget assumes $500 million is coming from the state. Is that going to happen? If you think so, please recall this headline from June 25, Rauner vetoes Illinois budget, cites $4 billion deficit. And curiously, that $500 million assumption is exactly the same as the amount that was to be borrowed from the pension plan. Chicago Public Schools Borrow $1 Billion to Fund Pensions Inquiring minds might even be wondering where the money came from to "fund" the pension this year. I use the words "fund the pension" very loosely. For the answer, please consider CPS Has 'No Choice' But To Borrow $1B Amid Cash Crunch. The school board approved the new borrowing on Wednesday, a day after the Illinois House rejected a three-week delay for a $630 million payment due to the teachers' pension fund on June 30. The district also is facing a $1 billion operating deficit. It might not sound like a good idea for CPS to borrow $1 billion more when it's already heavily in debt, but Performance Trust Capital Partners director Brian Battle said CPS has no real options. "They need to borrow this money short-term, so they can make a massive pension payment that's coming due here in two weeks. "CPS might get downgraded, because of an action like this, but really the rating agencies will deem this as a reasonable tactical thing to do. You know, you don't want to default if you have another option. You don't want to not make the payment if you have another option," he said. According to Battle, bond agencies want to see CPS come up with a long-term plan to address its budget and pension issues, and that might mean a property tax hike. Got That? The public schools borrowed $1 billion to make a $630 million payment due to the teachers' pension fund. The teachers then wanted to borrow $500 million of it right back, to support ongoing budget needs. Brian Battle, Performance Trust Capital Partners director, claims the rating agencies will view this as "reasonable" thing to do, even though Chicago's four pension plans have a combined $20.1 billion unfunded liability and funded ratios ranging from just 24% to 57%. Beware the Tax Man I commented on the tax aspect before, on May 4, in Beware, the Tax Man Has Eyes on You: Potential Hike for Illinoisans is Staggering. Short Synopsis: A report by Nuveen shows a pension payment spike looms in 2016, and the potential tax hike to fix it is staggering. Nuveen estimates property tax hikes of close to 50%  will be needed. Lost Cause Not a penny of taxpayer money should go to fund these lost causes. I find it hard to believe that Emanuel himself does not know the school system is truly bankrupt. To spare the citizens of Illinois massive tax hikes, the only reasonable course of actions are as follows: Halt defined benefit pension plans for new employees Eliminate collective bargaining of public unions Scrap Davis Bacon and all prevailing wage laws so that cities do not have to overpay for services Enact right-to-work legislation Pass bankruptcy legislation allowing cities, municipalities, and other taxing bodies the right to declare bankruptcy Had options 1-4 been done a decade ago, Illinois would not be as bad off as it is today. Now, even those measures cannot and will not fix the problems. Moody's Announcement "Chicago's Pension Pressures Will Grow For Years" On May 1, Moody's made this announcement: Regardless of legal and political outcomes, Chicago's pension pressures will grow for years. Illinois desperately needs bankruptcy legislation. 50% tax hikes are not only amazingly unfair, they will drive both corporations and individuals out of the state. Sad Conclusion Instead of admitting the obvious,  Emanuel expects $500 million from the State, nickels and dimes businesses to death, further makes Illinois and Chicago an uncompetitive place to do business, and even threatens massive property tax hikes. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

You are subscribed to email updates from Mish's Global Economic Trend Analysis To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States

All the latest news from people you know

Extra, extra! Read all about everybody's latest likes, Pins and follows   Get the app! Android · iOS

Discover all the latest finds   Vintage Design pinned 5 Pins Spiced Almond Butter Vintage Design Spread the Love.... Vintage Design Awesome Attic......   Other Pins they found     Vintage Design likes 3 Pins barbara bui Christine Rose E... Dreamy Shoes New puppy owner? Here are some great tips on How ... Cheryl Sousan | ... Everything for D...   Other Pins they found     Pinterest Facebook Twitter Google Blog   This email was sent to e0nstar1.blog@gmail.com. Don't want to receive this type of email? Unsubscribe. Have a question? Visit our Help Center. © Pinterest, Inc. 572 7th Street, San Francisco, CA, 94103 Privacy Policy | Terms and Conditions

Controlling Search Engine Crawlers for Better Indexation and Rankings - Whiteboard Friday - Moz Blog

Controlling Search Engine Crawlers for Better Indexation and Rankings - Whiteboard Friday

Posted by randfish

When should you disallow search engines in your robots.txt file, and when should you use meta robots tags in a page header? What about nofollowing links? In today's Whiteboard Friday, Rand covers these tools and their appropriate use in four situations that SEOs commonly find themselves facing.

For reference, here's a still of this week's whiteboard. Click on it to open a high resolution image in a new tab!

Video transcription

Howdy Moz fans, and welcome to another edition of Whiteboard Friday. This week we're going to talk about controlling search engine crawlers, blocking bots, sending bots where we want, restricting them from where we don't want them to go. We're going to talk a little bit about crawl budget and what you should and shouldn't have indexed.

As a start, what I want to do is discuss the ways in which we can control robots. Those include the three primary ones: robots.txt, meta robots, and—well, the nofollow tag is a little bit less about controlling bots.

There are a few others that we're going to discuss as well, including Webmaster Tools (Search Console) and URL status codes. But let's dive into those first few first.

Robots.txt lives at yoursite.com/robots.txt, it tells crawlers what they should and shouldn't access, it doesn't always get respected by Google and Bing. So a lot of folks when you say, "hey, disallow this," and then you suddenly see those URLs popping up and you're wondering what's going on, look—Google and Bing oftentimes think that they just know better. They think that maybe you've made a mistake, they think "hey, there's a lot of links pointing to this content, there's a lot of people who are visiting and caring about this content, maybe you didn't intend for us to block it." The more specific you get about an individual URL, the better they usually are about respecting it. The less specific, meaning the more you use wildcards or say "everything behind this entire big directory," the worse they are about necessarily believing you.

Meta robots—a little different—that lives in the headers of individual pages, so you can only control a single page with a meta robots tag. That tells the engines whether or not they should keep a page in the index, and whether they should follow the links on that page, and it's usually a lot more respected, because it's at an individual-page level; Google and Bing tend to believe you about the meta robots tag.

And then the nofollow tag, that lives on an individual link on a page. It doesn't tell engines where to crawl or not to crawl. All it's saying is whether you editorially vouch for a page that is being linked to, and whether you want to pass the PageRank and link equity metrics to that page.

Interesting point about meta robots and robots.txt working together (or not working together so well)—many, many folks in the SEO world do this and then get frustrated.

What if, for example, we take a page like "blogtest.html" on our domain and we say "all user agents, you are not allowed to crawl blogtest.html. Okay—that's a good way to keep that page away from being crawled, but just because something is not crawled doesn't necessarily mean it won't be in the search results.

So then we have our SEO folks go, "you know what, let's make doubly sure that doesn't show up in search results; we'll put in the meta robots tag:"

So, "noindex, follow" tells the search engine crawler they can follow the links on the page, but they shouldn't index this particular one.

Then, you go and run a search for "blog test" in this case, and everybody on the team's like "What the heck!? WTF? Why am I seeing this page show up in search results?"

The answer is, you told the engines that they couldn't crawl the page, so they didn't. But they are still putting it in the results. They're actually probably not going to include a meta description; they might have something like "we can't include a meta description because of this site's robots.txt file." The reason it's showing up is because they can't see the noindex; all they see is the disallow.

So, if you want something truly removed, unable to be seen in search results, you can't just disallow a crawler. You have to say meta "noindex" and you have to let them crawl it.

So this creates some complications. Robots.txt can be great if we're trying to save crawl bandwidth, but it isn't necessarily ideal for preventing a page from being shown in the search results. I would not recommend, by the way, that you do what we think Twitter recently tried to do, where they tried to canonicalize www and non-www by saying "Google, don't crawl the www version of twitter.com." What you should be doing is rel canonical-ing or using a 301.

Meta robots—that can allow crawling and link-following while disallowing indexation, which is great, but it requires crawl budget and you can still conserve indexing.

The nofollow tag, generally speaking, is not particularly useful for controlling bots or conserving indexation.

Webmaster Tools (now Google Search Console) has some special things that allow you to restrict access or remove a result from the search results. For example, if you have 404'd something or if you've told them not to crawl something but it's still showing up in there, you can manually say "don't do that." There are a few other crawl protocol things that you can do.

And then URL status codes—these are a valid way to do things, but they're going to obviously change what's going on on your pages, too.

If you're not having a lot of luck using a 404 to remove something, you can use a 410 to permanently remove something from the index. Just be aware that once you use a 410, it can take a long time if you want to get that page re-crawled or re-indexed, and you want to tell the search engines "it's back!" 410 is permanent removal.

301—permanent redirect, we've talked about those here—and 302, temporary redirect.

Now let's jump into a few specific use cases of "what kinds of content should and shouldn't I allow engines to crawl and index" in this next version...

[Rand moves at superhuman speed to erase the board and draw part two of this Whiteboard Friday. Seriously, we showed Roger how fast it was, and even he was impressed.]

Four crawling/indexing problems to solve

So we've got these four big problems that I want to talk about as they relate to crawling and indexing.

1. Content that isn't ready yet

The first one here is around, "If I have content of quality I'm still trying to improve—it's not yet ready for primetime, it's not ready for Google, maybe I have a bunch of products and I only have the descriptions from the manufacturer and I need people to be able to access them, so I'm rewriting the content and creating unique value on those pages... they're just not ready yet—what should I do with those?"

My options around crawling and indexing? If I have a large quantity of those—maybe thousands, tens of thousands, hundreds of thousands—I would probably go the robots.txt route. I'd disallow those pages from being crawled, and then eventually as I get (folder by folder) those sets of URLs ready, I can then allow crawling and maybe even submit them to Google via an XML sitemap.

If I'm talking about a small quantity—a few dozen, a few hundred pages—well, I'd probably just use the meta robots noindex, and then I'd pull that noindex off of those pages as they are made ready for Google's consumption. And then again, I would probably use the XML sitemap and start submitting those once they're ready.

2. Dealing with duplicate or thin content

What about, "Should I noindex, nofollow, or potentially disallow crawling on largely duplicate URLs or thin content?" I've got an example. Let's say I'm an ecommerce shop, I'm selling this nice Star Wars t-shirt which I think is kind of hilarious, so I've got starwarsshirt.html, and it links out to a larger version of an image, and that's an individual HTML page. It links out to different colors, which change the URL of the page, so I have a gray, blue, and black version. Well, these four pages are really all part of this same one, so I wouldn't recommend disallowing crawling on these, and I wouldn't recommend noindexing them. What I would do there is a rel canonical.

Remember, rel canonical is one of those things that can be precluded by disallowing. So, if I were to disallow these from being crawled, Google couldn't see the rel canonical back, so if someone linked to the blue version instead of the default version, now I potentially don't get link credit for that. So what I really want to do is use the rel canonical, allow the indexing, and allow it to be crawled. If you really feel like it, you could also put a meta "noindex, follow" on these pages, but I don't really think that's necessary, and again that might interfere with the rel canonical.

3. Passing link equity without appearing in search results

Number three: "If I want to pass link equity (or at least crawling) through a set of pages without those pages actually appearing in search results—so maybe I have navigational stuff, ways that humans are going to navigate through my pages, but I don't need those appearing in search results—what should I use then?"

What I would say here is, you can use the meta robots to say "don't index the page, but do follow the links that are on that page." That's a pretty nice, handy use case for that.

Do NOT, however, disallow those in robots.txt—many, many folks make this mistake. What happens if you disallow crawling on those, Google can't see the noindex. They don't know that they can follow it. Granted, as we talked about before, sometimes Google doesn't obey the robots.txt, but you can't rely on that behavior. Trust that the disallow in robots.txt will prevent them from crawling. So I would say, the meta robots "noindex, follow" is the way to do this.

4. Search results-type pages

Finally, fourth, "What should I do with search results-type pages?" Google has said many times that they don't like your search results from your own internal engine appearing in their search results, and so this can be a tricky use case.

Sometimes a search result page—a page that lists many types of results that might come from a database of types of content that you've got on your site—could actually be a very good result for a searcher who is looking for a wide variety of content, or who wants to see what you have on offer. Yelp does this: When you say, "I'm looking for restaurants in Seattle, WA," they'll give you what is essentially a list of search results, and Google does want those to appear because that page provides a great result. But you should be doing what Yelp does there, and make the most common or popular individual sets of those search results into category-style pages. A page that provides real, unique value, that's not just a list of search results, that is more of a landing page than a search results page.

However, that being said, if you've got a long tail of these, or if you'd say "hey, our internal search engine, that's really for internal visitors only—it's not useful to have those pages show up in search results, and we don't think we need to make the effort to make those into category landing pages." Then you can use the disallow in robots.txt to prevent those.

Just be cautious here, because I have sometimes seen an over-swinging of the pendulum toward blocking all types of search results, and sometimes that can actually hurt your SEO and your traffic. Sometimes those pages can be really useful to people. So check your analytics, and make sure those aren't valuable pages that should be served up and turned into landing pages. If you're sure, then go ahead and disallow all your search results-style pages. You'll see a lot of sites doing this in their robots.txt file.

That being said, I hope you have some great questions about crawling and indexing, controlling robots, blocking robots, allowing robots, and I'll try and tackle those in the comments below.

We'll look forward to seeing you again next week for another edition of Whiteboard Friday. Take care!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

You are subscribed to the Moz Blog newsletter sent from 1100 Second Avenue, Seattle, WA 98101 United States To stop receiving those e-mails, you can unsubscribe now.

Newsletter powered by FeedPress