Voting on websites is usually horribly broken and when it isn't broken it's pretty much useless. Sites that have voting usually end up as echo chambers. Also on sites where the worthless points are tracked people with a lot of points often become moderators or their opinions are more important than those of other commenters and get more points for them just because they are popular.
To me upvote is probably the worse. What makes upvote bad is that even though it leads to an echo chamber and makes the site sort of a popularity contest it is still seen as a positive thing and people feel positively when they give upvotes. Upvotes are also often given when you simply agree with the viewpoints of the other person, the quality or if what they say is true does not really matter.
Upvotes however aren't always bad. I do think that they can be useful especially for the topics but then it is really easy to bury other interesting threads. Instead people should just learn to look at a time line view of submissions or threads and not just front pages.
Downvotes also have the problem of people voting because of their opinion when they should be used to judge the quality of the comment but I don't see this as the biggest problem with downvoting. I think that downvoting discourages discussion, instead of having to tell the other person that they are wrong and why you only have to click downvote. Even if someone's post only says "You're wrong." it is still better than a downvote as you can see who said it and you can ask for clarification.
There is also another issue with most systems that have downvotes (Hacker News*, Reddit, Lobsters, Slashdot, ...) which is that the comments that are downvoted are hidden. Because downvotes are often cast because you disagree this very easily leads to an echo chamber and possibly even ultimate silencing of the opposing views. The comments that people disagree should instead rise to the top as they usually generate more and more interesting discussion than when people all agree with each other.
If you are thinking of starting a new site for discussion please take a moment to consider how you could do it without using a system based on votes. If you really want to allow comments then at least limit it in some way, one way I could think of is letting people upvote only comments they have replied to.
* Note that Hacker News tries to actively block Tor users. Their system however is imperfect and can be circumvented by trying a few different circuits. I like using arm** for controlling my Tor.
** Will be renamed to nyx in the future, if you see that this link is broken then I would appreciate it if you notify me in some way.