Most gear is pretty straightforward to select: pick whatever has the highest stamina, agility and hit / exp depending on your needs. Trinkets are a different matter however, since they almost always have just two wildly different effects (one passive stat and one proc or clicky) you can form many combinations with the two trinket slots.
I will post a list of viable tank trinkets (mostly from recent raids) and how I feel about them. Although I will try to provide some extra numbers and analysis when possible, the matter of trinket selection remains highly subjective and depends greatly on the circumstances: Your gear, the composition of your raid and the boss you're facing. Note that everything is written from the point of view of a Druid, but most of it also applies to the other tank classes.
A short note on armor and stamina as contributors to EH
You'll find that alot of trinkets feature armor and/or stamina and both stats contribute to your "Effective Health", which is the amount of damage (before mitigation) you can take in a worst-case scenario without dying. Why this measure of Effective Health is inadequate, the reader is referred to an earlier blog-post I made.
However, assuming the naive definition of Effective Health, one can calculate the contribution of 1 point of armor versus that of 1 point of stamina. This is fairly straightforward:
R(a) = a / (a + 467.5 * l - 22167.5)
R(a) is the damage reduction from armor expressed as a fraction of 1, a is your armor-value and l is the level of the attacker (83 for bosses). We then find that:
EH(a,h) = h / (1 - R(a))
Or: Your effective health (EH) as a function of armor (a) and health (h) is your health divided by one minus your damage reduction from armor.
You can differentiate this function to a and h and find the respective contributions to EH from 1 point of armor and 1 point of health. Then you just have to work out how much health you gain from 1 point of stamina (answer: 16.35 for a Feral Druid with BoKings) and you get the final numbers. For the further discussion, I will use 66K HP and 35K armor as numbers. The scaling factors are not massively different if you change these base numbers a bit. None of the Druid talents and abilities change the armor on trinkets. The Austere Earthsiege Diamond (meta gem) does, but it adds only 2%, which doesn't really affect the balance between armor and stamina.
1 point of armor = 3.97 "EH" (4.05 with meta gem)
1 point of stamina = 50.76 "EH"
1 point of stamina = 12.79 points of armor (12.54 with meta gem)
Again, I'd like to emphasize that this analysis overrates the value of stamina in almost all fights. Additionally, the added benefit of armor in that it reduces the mana-drain on healers is completely ignored.
Darkmoon Card: Greatness (DMF cards, BoE) - [Passive: 90 agi] [Proc: Dealing damage, 35% chance, +300 agi for 15 sec, 45 sec ICD] - Despite being low-level, this one deserves an honorable mention. Agility isn't the best stat, but it beats dodge-rating thanks to the contributions to armor and crit as well as the scaling with BoKings. With an uptime between 25% and 30%, the proc is pretty good. It doubles as DPS-trinket, making it an excellent purchase for a new Feral Druid.
The Black Heart (ToC5 normal) - [Passive: 126 sta] [Proc: Hit by melee attack, 25% chance, +7056 armor for 10 seconds, 45 sec ICD] - The Black Heart is surprisingly good for its item-level. The proc is rather strong, but can't be relied on of course. It's also very easy to get, as ToC5 normal can be farmed repeatedly making it a very good starter option, but one that you can safely stick with for a while if your luck with trinkets is bad.
Eltrigg's Oath / Fervor of the Frostborn (ToC10) - [Passive: 114 (H: 126) dodge] [Click: +1265 (H: 1422) armor every time an attack hits you, stacks to 5, entire effect lasts 20 sec, 2 min CD] - This trinket is one of least interesting. Avoidance is weak and WotLK and due to the ramp-up-time, the clicky-effect isn't "there when you need it", like a clicky-effect should. The Darkmoon Card: Greatness beats this one in almost every way.
Ick's Rotting Thumb (PoS heroic) - [Passive: 113 dodge] [Click: +4104 HP, 15 sec, 3 min CD] - Just like the ToC10 trinket, this one is pretty weak. It's only saving grace is the clicky-effect, which is not bad.
Juggernaut's Vitality / Satrina's Impeding Scarab (ToC25) - [Passive: 192 (H: 216) sta] [Click: +4610 (H: +5186) HP, 15 sec, 3 min CD] - The normal mode version is pretty much the baseline trinket for tanks these days. ToC25 can be pugged quite effectively and this trinket shouldn't be that hard to get. At ilvl 245 it's unmatched and you won't find upgrades until you go to higher item levels. The heroic version is even more desirable, but hard to get now that most guilds ignore ToGC in favor of ICC.
Glyph of Idomitability (50 EoT) - [Passive: 1792 armor] [Click: +512 dodge, 20 sec, 2 min CD] - The armor corresponds to 143 stamina in the naive EH model (so in reality it's more!), which makes it rather strong. Unfortunately, the clicky-effect is pretty weak: When you use a clicky, you usually do it because you're at risk of dying and you want something solid to save you. And dodge just doesn't cut it there. It's easy to get though and I think it beats all lower-level trinkets.
Unidentifiable Organ (ICC10) - [Passive: 1890 armor] [Proc: Hit by melee attack, 60% chance, +24 stamina for 10 seconds, stacks to 10] - The armor corresponds to 151 stamina in the naive EH model (so in reality it's more!). The proc is quite controversial and its value greatly depends on what you're fighting. Some people claim the trinket is crap, since the stamina proc is so random, but they forget that the main benefit of the trinket is the armor and the proc is just icing on the cake. Overall, it's just another random proc trinket, but it's one of the better random procs. You'll have some stacks up most of the time. Obviously it's best suited for fights where you get hit alot. Consequently, it's not very useful outside ICC, where you avoid 20% more and bosses attack less often.
Corpse Tongue Coin (ICC25) - [Passive: 152 (H: 172) dodge] [Proc: Melee hit that takes you below 35%, +5712 (H: 6426) armor for 10 sec, 30 sec ICD] - The idea of a proc that's there when you need it the most is quite good. Unfortunately, its execution leaves much to be desired. Using the numbers I mentioned earlier (66K HP, 35K armor) and assuming the hit that procs the trinket lands you at just below 35%, the armor proc will let you take 2555 more damage before dying (assuming no healing). In reality, the hit that procs the trinket might land you anywhere between 0% and 35% reducing the effectiveness of the proc. Furthermore, magical burst does nothing unless it's directly followed by a melee swing (that doesn't outright kill you). The passive dodge on top of the weakish proc makes this trinket not worth getting.
Corroded Skeleton Key (60 EoF) - [Passive: 228 stamina] [Click: 6400 damage absorbed, lasts 10 sec, 2 min CD] - The de-facto successor of JV / SIS from ToC25. The clicky can be seen as a very beefy healthpot and can be nicely combined with a pot and a healthstone for a big HP boost in emergencies. This one should be high on your shopping list unless you already have two good ilvl 258+ trinkets.
Sindragosa's Flawless Fang (ICC25) - [Passive: 228 (H: 258) stamina] [Click: +239 (H: 268) resists for 10 sec, 1 min CD] - The clicky of this trinket is very situational: It's excellent for bosses that have large magical bursts (Ironically, Sindragosa, the boss that drops it, is a prime example), but it's rather weak if the magic component of the damage only comes from minor AoE effects. The stamina makes it a good competitor for the CSK though.
So what would I use?
Right now, I use CSK and normal mode Organ for most of my tanking. I value the Organ over normal-JV (not had much luck with heroic-JV), but I swap Organ out for JV on fights with alot of magic burst: Primarily Sindragosa. I've passed on the Corpse Tongue Coin a couple of times, it's simply not worth it. I also have the heroic Eltrigg's Oath in my bags, but it's only collecting dust.
Ideally, I would probably use the heroic Organ with heroic Sindragosa's Flawless Fang. However, sticking to normal mode loot, my preferred combo would be Organ + CSK which I use now for regular fights. On fights with alot of magic, I'd like to use SFF + CSK.
For a starting tank, I would strong recommend The Black Heart, it's easy to farm and can last you a long time. In addition, pick up the Glyph of Indomitability. You can replace The Black Heart by the CSK if you have enough EoF (and nothing else you'd rather buy with them). In the meantime, keep running ToC25 for JV / SIS.