Correct spelling, explanation: the word subcategories is a combination of two parts: the prefix sub- and the plural noun categories. The first one is a word-forming element meaning under, beneath, behind. The second one originates from French catégorie, from Late Latin categoria, and from Greek katēgoria. As you can see, the etymology indicates that the third vowel in this word should be -e, not -a. You have to remember that when spelling it. Subcatagories is just a plain mistake.

Definition of subcategories:
noun (plural), a secondary or subordinate category
Each category is broken down into six subcategories.
Scientists can now classify tumors into smaller subcategories.

Collocations with subcategories:
Some most commonly used collocations include: subcategory of and fall into a subcategory.
Computer viruses are divided into three broad categories and each of them falls into subcategories.
I know some subcategories of plants.

Incorrect spelling, explanation: there is no word like subcatagories in any dictionary. The right version of this noun – subcategories – contains only one -a letter. Its spelling may be tricky due to a large number of vowels. If you want to be absolutely sure how to write down a word, always check its etymology. In this case, you should know that subcategories consist of two parts: sub- and categories. The second one comes from French, Late Latin, and Greek. You can’t spell it differently than subcategories.