This post doesn't refer to every single person who holds the Bible inerrant and infallible. It's simply written in reaction to the behavior of some who hold that view.
There are a variety of reasons why I don't hold the Bible to be 'without error.' I don't mean that the writers deliberately lied -- at least not as that time-frame would refer to as 'lie.' But they did lack a lot of knowledge in terms of science and just the universe in general.
Anyway, one of the other reasons is because very often, it comes across as though inerrants don't listen to those who aren't Christian: and this pretty much covers ever area. Those in another religion, those who are just spiritual, those who are agnostic/atheists (either were something else and become that or used to be Christian and are now that or those who were just always that), those who don't believe every aspect of conservative Christianity.
A very common reaction to those who aren't Christian is that they're actively rejecting God and blatantly living in sin. And no matter what the response is, the fundamentalist inerrantist )shorted to FI) rejects your answer, for the FI knows you better than you know yourself. Essentially, the FI isn't listening to anything you say, but rather interpreting you through the Bible.
The catch to that answer is twofold: 'actively rejecting God.' One, that assumes that everyone is agreeing on the concept of God. Now, for me, if God is defined Biblically as Spirit, Light, and Love, among other adjectives, then actively rejecting that concept of God becomes incredibly difficult. Most people gravitate towards the light. Many prefer to live a life of love, and if dwelling in love means dwelling in God ... then don't most people dwell in love? A friendship or a marriage or being a parent -- those do require the greatest type of love there is, a self-sacrificial love.
The second 'fold' is blatantly living in sin. Most of what Paul lists as not the fruit of the Spirit are feelings/actions that most people don't enjoy having: adultery, hatred, wrath, strife (probably even idolatry, because part of worshiping something that's unhealthy is the power it holds over you). Now, if living in those feelings while in the moment, one can feel perfectly justified. But for the most part, when someone no longer hates a brother, or lusts after someone/something, there's a tremendous amount of joyful freedom. So even 'blatantly living in sin' really doesn't fly, when I consider most people I come in contact with.
If the rebelling-against-God/blatant sin isn't the response, then the answer is usually that the non-FI doesn't know how to correctly read the Bible, or is eager to look for reasons why to discount the Bible, or trying to force God to work according to someone's terms. But that again gets lumped into the 'not listening' category. For many people, trying to work the Old Testament events around a all-loving, infinite, all-knowing, all-powerful God (or just trying to work it around how Jesus claims God is) doesn't work. Israel had a tribal mindset for much of the Old Testament -- thus, God is portrayed with tribal God behavior. I truly believe that if that type of behavior were shown to belong to an infinite God in *any* other setting, it would be enough to discount the Old Testament as divine/infinite. But it's part of an "inerrant, infallible" text, and so must be true. To a non-FI, or even a non-inerrantist, that comes across as an excuse, and that too many things have to be twisted into reasons why the Bible is inerrant. To a FI, the other person is simply hiding in sin, or unwilling to face sin, or ... something. Which brings us back to the non-listening aspect.
Another reason why I don't hold to inerrancy is that it can be an incredibly dangerous position. In the past, the Bible has been used to justify slavery, acts such as the Inquisition, and holding women to a second-class citizenship. Today, most consider all of those to be immoral acts, and would say that all occurred through a mis-reading of the Bible. That's fine. But all those acts were, in general, committed by people who held the Bible to be without error, and to act otherwise was to go against the word of God. If that was the case even 150 years ago, what is being held today as 'without error' that is actually just as immoral? Even the argument that reading the Bible while humbly asking God to do His will will lead to a moral act doesn't work, because those who supported slavery did believe that they were following God's Will -- and to fight against slavery was to be in rebellion against God.
So ultimately, that is why I don't hold the Bible to be inerrant or infallible. Because if I did, it would strip me of my ability to listen, or to understand why someone does not follow the Bible. It would be much to easy for me to lump anyone who is not like me into a category to be dismissed -- and that's already too easy to do most days. It could also run the risk of stripping my ability to make moral judgements. That can come across as arrogant, yes, but I've seen all the damage inerrency has done, historically. I do believe good can come from the Bible -- I've had good come from the Bible. I know inerrants online who are devout Christians who would never do anything immoral, and do understand why others have problems with Christianity. It just seems that they are on the rare side, too often.
It's simply important to 'test the spirits,' and consider the cultural context in which the Bible was written.