Central Perk

joi, 5 decembrie 2013

6 Steps to Finding all Your Website URLs

6 Steps to Finding all Your Website URLs

Posted: 04 Dec 2013 06:59 AM PST

Removing pages from your website? Going through a site redesign or migration but not sure you have all the URLs on your website? It's an issue that we will all go through at some point in our SEO life. I have been involved in a lot of projects recently where I've needed to find all the URLs that were on a website, and it can be a pain to do! However, I've now managed to get it down to 6 easy steps, and I wanted to share them with you.

Step 1. Crawl your website

This is an obvious one in my opinion. Looking at your website and gathering all the URLs that you can find should be easy right? Well, maybe if you only have a 10′s of pages, but if you are at enterprise level then this isn’t so easy.

Most of you will already know that you can use tools such as Xenu and ScreamingFrog, but if your site is at enterprise level, these may not be robust enough. In this case you could turn to DeepCrawl, which specialises in crawling large websites.

Once you have run the crawl, place those URLs into a spreadsheet on a tab labelled ‘Website Crawl’. You will also want to start a ‘Master List’ so that you have a single URL list, so go ahead and create that too. We will be constantly adding to this list as we go through the steps.

Step 2. Visit your Analytics Package

Reviewing your analytics data is extremely important for all levels of marketing, but many people don’t realise that this is a good place to see what pages you have on your website.

We are looking for all the pages on the website, so you need to head over to Content or Behaviour in your analytics package, change the date so you have a range of at least 18 months, and hit the download button. If you have too many URLs for the download or it is taking a considerable amount of time, you may want to investigate using the API.

Once you have these URLs create a new tab called ‘Analytics Data’ and place your URLs here, you then want to add a copy to the bottom of the list in the ‘Master List’ tab. Now to step 3.

Step 3. XML Sitemaps

This is another place that is commonly forgotten when looking for URLs. The XML sitemap should, ideally, be the place to find the most up to date version of all URLs for your website. After all, it is the place you are asking the search engines to look to help improve your visibility.

Luckily, with an XML file you can open it straight into Excel, so do that now. You will need to do some formatting to remove the unnecessary tags that accompany the XML sitemap, but once you've done this it will leave you with a list of URLs. Copy this list into your original spreadsheet onto a worksheet called ‘Sitemaps’, and then copy it into the ‘Master List’ tab.

Step 4. Get a list of Your Most linked pages

Everyone likes a good link right? So you wouldn’t want to do anything that may lose an influential link. The next step is to download a list of the URLs that are your most linked to pages. You will probably need to do this from multiple tools if you have access to them, but at the very minimum you should download a list of URLs from GWT.

Tools to download most linked to pages from include:

Google Webmaster Tools
Bing Webmaster Tools
Open Site Explorer Top Pages
MajesticSEO

Once you have these URLs, collate them and add them to a new tab called ‘Most Linked Pages’, and add a copy to the bottom of the URL list on the ‘Master List’ tab. Now move on to the fifth step.

Step 5. Scraping the SERPs

So far. we have used numerous tools to get all our URLs, but we haven't checked the search engines! So let's do this now. You will need a scraping extension for your browser. I use Scrape Similar for Chrome.

Go to the search engine of choice, you will want to check at least Google & Bing in the UK, and type in site:domain.com. Now, change the settings of SERP page to show 100 results (we want to do this as quickly as possible!) – this can be done by going to your account settings and changing the view from 10 to 100. You may also need to remove Instant Search from the tick box options.

Now you should be able to see 100 results from your domain. Hover over the first result title tag, right click and select "scrape similar". This should bring up another dialog box with the list of the URLs from the first 100 SERPs and provide you with the option to either put it straight into excel or Google Drive. Either option is good at this point. You will need to go through all the listings that the search engines have returned – this could take a bit of time! There might be a quicker way to do this, and if you know one I would be happy to hear about it in the comments below.

Once you have gone through the results and collated the URLs, put them in a new tab called ‘SERP Scrapped URLs’ and add the list to the bottom of the URLs you have gathered from Steps 1-4 in the ‘Master List’ tab.

Step 6. De-dupe & Check

Wow, you have come a long way and more than likely have a lot of URLs within your spreadsheet. Most of those are likely to be duplicated, at least we hope they are as it will mean you are doing a good job. In Excel there is a feature that allows you to remove all duplicates and leave you with a unique list of URLs. This feature is found in Data > Remove Duplicates. Go ahead and do this.

Hopefully this will leave you with a good amount of URLs. Now for the final step, copy the list of URLs and run them through a crawler, I’d use ScreamingFrog to allow you to check the HTTP status of those URLs. Now you have the status codes, copy this list back into your spreadsheet, which will leave you with a list of as complete as possible URLs with status codes. Now you are done!

If you have completed all six steps, then you should have a pretty thorough list of the URLs that are located on your website. I hope this was helpful and provides some structure to finding all the URLs that you need. Have I missed anything out? Is there a quicker more reliable way of getting all the URLs? I would love to hear your thoughts in the comments below or over on twitter @danielbianchini.

The post 6 Steps to Finding all Your Website URLs appeared first on White Noise.

Seth's Blog : The moderation glitch

The moderation glitch

More doesn't scale forever. Why are we so bad at enaging with this obvious truth?

In Malcolm's new book, he points out that our expectation is that most things will respond in a linear way. More input gets us more output. If you want a hotter fire, add more wood. If you want more sales, run more ads.

In fact, it turns out, most things don't respond in a linear way. It's more of a steep curve (he calls it an inverted U). For a while, more inputs get you more results, but then, inevitably, things level off, and then, perversely, get worse. One brownie makes you happy, a second brownie, maybe a little more. The third brownie doesn't make us happy at all, and the fourth brownie makes us sick.

Health care is a fine example of this. First aid makes a huge difference. Smart medical care can increase our health dramatically. But over time, too much investment in invasive medicine, particularly at the end of life, ends up making us worse, not better. Or, in a less intuitive example, it turns out that class size works the same way. Small classes (going from 40 to 25 in the room) make a huge difference, but then diminishing class size (without changing teaching methods) doesn't pay much, and eventually ends up hurting traditional classroom education outputs.

But here's the unanswered question: if the data shows us that in so many things, moderation is a better approach than endless linearity, why does our culture keep pushing us to ignore this?

First, there are the situations where one person (or an organization) is trying to change someone else. Consider the high-end omakase sushi bar, where, for $200, you're buying a once-in-a-lifetime meal. The chef certainly has enough experience to know that he should stop bringing you more food, that one more piece of fish isn't going to make you happier, it's quite likely to make you uncomfortable. But he doesn't stop.

Or consider the zero-tolerance policy in some schools. We know that ever more punishment doesn't create better outcomes.

Here's the problem with the inverted U: We aren't certain when it's going to turn. We can't be sure when more won't actually be better.

As a result of this uncertainty, we're likely to make one of two mistakes. Either we will stop too soon, leaving stones unturned, patrons unsatisfied, criminals unpunished... or we will stop too late, wasting some money and possibly missing the moderation sweet spot.

You already guess what we do: we avoid the embarrassment of not doing enough. The sushi chef doesn't want someone to say, "it was great, but he wasn't generous." The politician says, "I don't want any voter to say that even one criminal got away because I was soft on crime."

We always start with intent, as Omar Wassow has pointed out. It's intent that gets us to take action and to start marketing and spending. But intent and results are different things.

We market our solution (to ourselves and to others) and that marketing drives our actions. As long as we're uncertain as to where the curve turns, we're going to have to push that marketing message forward. It's a lot more difficult to sell the idea of moderation than it is to sell the earnest intent of joy or punishment or health or education.

Moderation is a marketing problem.

(this is getting long, sorry, but I hope it's worth it)

The other category of interventions are the things we do to ourselves. This is the wine drinker who goes from the health benefits of a daily glass of wine to the health detriments of a daily bottle or two. This is the runner who goes from the benefits of five miles a day to knees that no longer work because he overdid it.

Here, the reason we can't stop is self marketing plus habit. Habits are the other half of the glitch. We learn a habit when it pays off for us, but we're hardwired to keep doing the habit, even after it doesn't.

Hence the two lessons:

1. Smart organizations need to build moderation-as-a-goal into every plan they make. Every budget and every initiative ought to be on the look out for the sweet spot, not merely "more." It's not natural to look for this, nor is it easy, which is why, like all smart organizational shifts, we need to work at it. How often does the boss ask, "have we hit the sweet spot of moderation yet?"

If doctors were required to report on quality of life instead of tests run, you can bet quality of life would improve faster than the number of tests run does.

2. Habits matter. When good habits turn into bad ones, call them out, write them down and if you can, find someone to help you change them.

"Because it used to work," is not a sensible reason to keep doing something.

[But please! Don't forget the local max.]

• Email to a friend •

miercuri, 4 decembrie 2013

Mish's Global Economic Trend Analysis

Video "Forget About The Price Tag" Wins Paltry $2000 HHS Grand Prize for Promoting Obamacare; What's Next for the Winner?
Number of Banks and Percentage of Problem Banks Over Time
Taxed to the Point of No Recovery; France Plans Tougher "Exit Tax"
Government About to Destroy American Mortgages Permanently Warns Dick Bove; Mish Says Nonsense

Video "Forget About The Price Tag" Wins Paltry $2000 HHS Grand Prize for Promoting Obamacare; What's Next for the Winner?

Posted: 04 Dec 2013 06:07 PM PST

The Daily Caller reports Video called 'Forget About The Price Tag' wins HHS grand prize for promoting Obamacare.

The Department of Health and Human Services has crowned a YouTube video entitled "Forget About The Price Tag" as the grand prize winner in a contest meant to encourage young people to sign up for Obamacare.

The video contest, announced in August — in partnership with a group called Young Invincibles — encouraged participants to produce clips filled with pro-Obamacare messaging.

HHS's grand prize-winning video, announced Monday by the White House, features a young woman named Erin McDonald singing an Obamacare-loving version of Jessie J's hit single, "Price Tag."

Video of Erin McDonald

"You are young and wild and free but you need to stay healthy .... Ain't about the, uh, cha-ching cha-ching. Ain't about the, yeah, bla-bling bla-bling. Affordable Care Act. Don't worry 'bout the price tag."

If the video does not play, click on the top link.

What's Next for the Winner?

For her efforts Erin McDonald wins a paltry $2000. I am quite sure Mish readers could come up with far more than that to get her to sing "Audit the Fed".

Alas, that can never happen.

Erin has a beautiful voice and just won the political jackpot and probably an invite to the WhiteHouse.

More importantly, I bet she gets a multimillion dollar contract from some record company to sing whatever the heck she wants (and I highly doubt it's about Obamacare or the Fed).

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com

Number of Banks and Percentage of Problem Banks Over Time

Posted: 04 Dec 2013 12:53 PM PST

Most understand the "too big to fail" banks have gotten bigger and bigger over time as they swallow up smaller banks bit by bit.

Here is the concept in picture form courtesy of reader Tim Wallace. Click on any chart for sharper image.

Number of Banks Over Time by Type

Total Number of Banks

Percent of Banks Vs. 1990 Total

Problem Banks as Percentage of All Banks

Anyone feel any safer by this?

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com

Taxed to the Point of No Recovery; France Plans Tougher "Exit Tax"

Posted: 04 Dec 2013 11:17 AM PST

In a feudal as well as futile attempt to keep wealthy French citizens from leaving the country, France hikes the "Exit Tax" on transfers of wealth to outside of France. They also lower the base and increase the number of things on which the tax applies.

According to a "pay-walled" article on Le Monde of which I can only read a part ... "The exit tax was established in 1999, repealed in 2005, then reintroduced in the first Amended Finance Act for 2011. The law was intended to limit the temporary exile of entrepreneurs wanting to sell their stakes in more favorable tax conditions than under domestic law."

Reader Bran informs me the the article states they plan to integrate collective investment in realty into the realm of the exit-tax.

Taxed to the Point of No Recovery

Here are some pertinent points on exit taxes and taxes in general by Veronique de Rugy writing for the National Review: France to Beef Up Its Exit Tax.

The French government seems committed to taxing itself beyond the point of no recovery. You've heard me talk about how over the years, and in particular over the last four years, France has relied heavily on tax increases in trying to contain its huge deficits. Everyone knows about how President Hollande campaigned for and then proposed a 75 percent tax rate on personal income above €1 million.

One aspect of France's confiscatory taxes that's often overlooked by Americans is that previous President Nicolas Sarkozy was almost as bad as Hollande when it came to raising taxes. In fact, data compiled by taxpayers' watch groups and newspapers show that between 2007 and the end of 2012, taxpayers were subjected to 205 separate increases in their tax burden, from excise levees on televisions, tobacco, and diet sodas to multiple increases in the capital taxes and a wealth-tax hike. Sarkozy is also responsible for increasing the top marginal income tax rate from 40 to 41 percent in 2010, and again, to 45 percent, in 2012.

Le Monde published a special report in September 2013 in which the liberal newspaper used data from the Ministre des Finances to show that, since 2009, under both Presidents Sarkozy and Hollande, 84 new taxes have been instated. The article also notes that Sarkozy increased tax revenue by €16.2 billion in 2011 and €11.7 billion in 2012, while Hollande added another €7.6 billion on top of that as soon as he was elected. He's planning to raise an additional €20 billion in 2013. That's €55.5 billion in new tax revenue in four years, with more than half of the total collected from businesses.

And there's more: The French government has also announced that it will beef up the exit tax, a tax first implemented by Sarkozy in 2012 intended to slow the pace of people leaving the country for tax reasons. The exit penalty taxes capital gains at the rate of 19 percent and adds a 15.5 percent payroll-tax-like penalty. The tax isn't paid as taxpayers exit the country, but people have to pay the tax if they sell their assets within eight years after their exit.

Tax Policy Theory and Results

Tax News reports France Plans Tougher 'Exit Tax'

The French National Assembly Finance Committee has adopted an amendment to the country's 2013 year-end supplementary finance bill, toughening the so-called "exit tax."

Since March 3, 2011, French taxpayers with wealth in excess of EUR1.3m, electing to transfer their fiscal residence abroad, are subject in France to a tax on latent capital gains crystallized at the time of their departure, if they cede the assets within eight years.

Significantly tightening the existing provisions, the adopted parliamentary amendment provides that the threshold for application of the levy should be lowered to EUR800,000.

Furthermore, the measure stipulates that the tax should be due if taxpayers cede their assets within 15 years following their expatriation, rather than eight.

Despite the tough stance, the measure is expected to have very little impact on the public finances. Last year, the exit tax served to yield a meagre EUR53m for the state.

French Flee a Nation in Despair

Inquiring minds may wish to consider an excellent article on flight from France on the Telegraph referenced by the National Review: Down and Out: the French Flee a Nation in Despair. Here is the opening statement:

The failing economy and harsh taxes of François Hollande's beleaguered nation are sending thousands packing - to Britain's friendlier shores.

By 2014, France's public expenditure will become the world's highest, at 57 per cent of GDP Photo: Howard McWilliam

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com

Government About to Destroy American Mortgages Permanently Warns Dick Bove; Mish Says Nonsense

Posted: 04 Dec 2013 01:03 AM PST

Citing Dick Bove, Yahoo!Finance reports Government About to Destroy American Mortgages Permanently.

Mortgages as we know them are going away in the next four years, warns Dick Bove, vice president of research at Rafferty Capital. Bove, one of the most widely-respected banking analysts in the world, is certain that will have devastating consequences for housing and the rest of the American economy.

The removal of the two most important players in American mortgages – the Federal National Mortgage Association ("Fannie Mae") and the Federal Home Loan Mortgage Corporation ("Freddie Mac") – threatens the very foundation of the American economy, according to Bove.

These two government-sponsored entities – along with the smaller Government National Mortgage Association ("Ginnie Mae", a government corporation that broke off from Fannie Mae) – issued 98% of the $1.4 trillion in mortgage-backed securities in the United States so far in 2013. These securities are sold in order to add liquidity to the mortgage market, thereby making funds available to borrowers.

"If Fannie and Freddie go away, what then happens to the mortgage markets?" asks Bove. "The answer to that question is that we no longer have things like 20-year and 30-year mortgages because banks are not going to put that type of mortgage on their balance sheets. And we won't have fixed-rate mortgages."

Bove says the banks he spoke with won't be able to provide 30-year mortgages in large quantities without Fannie Mae and Freddie Mac in the markets. "I've called a number of very large banks – the largest issuers of mortgages in the United States – and asked them, 'If there was no Fannie and Freddie, what would be the typical mortgage in the United States?' And, the answer is a 10- to 15-year adjustable rate mortgage."

The end of Fannie Mae and Freddie Mac is a major sea-change in how the government views affordable housing, according to Bove.

"It is no longer the goal of the United States government that every household should have its own home," say Bove. "In my view, that's a call for a return of public housing and all of the ills that went with public housing."

Affordable Housing Nonsense

The results of "affordable housing" programs speak for themselves.

In the last decade, hundreds of "affordable housing" programs at the federal and state level did anything but make housing affordable.

Together with president George Bush's inane "ownership society", the price of homes skyrocketed, as did taxes on homes. And cities did not use those tax dollars very wisely, did they?

Low interest rates, declining lending standards, ownership promotion mentality, 95% mortgages, and a host of other silly ideas fueled the biggest housing bubble in history.

Then when housing prices crashed, the Fed and government bureaucracies at every level (city, state, federal) acted in unison, hoping to force home prices back up.

So spare me the sap about "affordable homes". Neither the Fed nor government bureaucrats really want "affordable housing".

I highly doubt Dick Bove does either. But if by some miracle he does, he sure as hell does not know the best way to achieve that goal.

Oh The Horror

Bove laments "If there was no Fannie and Freddie, what would be the typical mortgage in the United States?' And, the answer is a 10- to 15-year adjustable rate mortgage."

If Bove is correct, that would be a great thing! People would not over-leverage, prices would be stable, and at the end of 10 years people would actually "own" something.

California Commercial Banker Chimes In

A California Banker friend (ACB) sent me the above link and also chimed in with his thoughts.

Hi Mish

I'm sure you've written before about closing down Fannie and Freddie. I too support the idea. In essence, we should return lending to the free market. The government sponsored lending boom via GSEs, together with cheap money from the Fed and declining lending standards, led to artificially high real estate values culminating in various bubbles.

Bove's research with bank executives leads to the conclusion that all mortgages in the future will be 10-15 year loans on variable rates. I find that odd, because I've been a Commercial Banker for 20 years, we write variable rate commercial real estate loans on a 25 year amortization due in 5, 7 or 10 years all the time.

I do agree that most mortgages will be variable/adjustable, as a bank you just can't take the interest rate risk by offering longer term fixed rates if you hold those loans on your balance sheet.

Another possibility would be something along the lines of bonds (not government backed) to support some fixed rate lending. The underwriting criteria behind these loans might be and should be stronger, say a minimum 25-30% equity, 28% front end debt/income ratio, prudent back end ratios, upper tier credit history, and strong job history. In essence, left to its own accord, the market would rid itself of the flimsy underwriting under the old Freddie and Fannie model.

Then, if Freddie and Fannie went away, wouldn't major banks who write most of the mortgage loans have to pay a little better interest rate on certificate of deposits (benefiting seniors) to attract capital into the banks to make mortgage loans?

We've become so government dependent, we fail to understand the free market will solve the alleged mortgage problem quite easily.

Lending should be a prudent thing, not a government sponsored free-for-all for political purposes.

Thanks,
California Banker

Bingo.

ACB and I welcome a return to lending sanity and an end to boom-bust cycles sponsored by the Fed and government bureaucrats.

Bove believes government can and should promote "affordable housing" even though history (and common sense) suggest the idea is ridiculous.

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com